[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 387061388 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.003226] x2apic enabled [ 0.004003] Switched APIC routing to physical x2apic. [ 0.005006] kvm-guest: setup PV IPIs [ 0.008311] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009009] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010004] pid_max: default: 32768 minimum: 301 [ 0.011087] LSM: Security Framework initializing [ 0.012026] Yama: becoming mindful. [ 0.012899] SELinux: Initializing. [ 0.013037] *** VALIDATE selinux *** [ 0.020313] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.024435] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.025087] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.026065] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027074] *** VALIDATE tmpfs *** [ 0.028345] *** VALIDATE proc *** [ 0.029145] *** VALIDATE cgroup *** [ 0.030004] *** VALIDATE cgroup2 *** [ 0.031194] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.032097] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.033003] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.034016] Spectre V2 : User space: Vulnerable [ 0.035003] Speculative Store Bypass: Vulnerable [ 0.037520] debug: unmapping init [mem 0xffffffffb5259000-0xffffffffb5260fff] [ 0.039165] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.040463] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.041010] ... version: 2 [ 0.042006] ... bit width: 48 [ 0.043005] ... generic registers: 4 [ 0.044009] ... value mask: 0000ffffffffffff [ 0.045011] ... max period: 00007fffffffffff [ 0.046006] ... fixed-purpose events: 3 [ 0.046927] ... event mask: 000000070000000f [ 0.047260] rcu: Hierarchical SRCU implementation. [ 0.049126] smp: Bringing up secondary CPUs ... [ 0.050380] x86: Booting SMP configuration: [ 0.051017] .... node #0, CPUs: #1 #2 #3 [ 0.054010] smp: Brought up 1 node, 4 CPUs [ 0.056006] smpboot: Max logical packages: 1 [ 0.057012] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.145013] node 0 deferred pages initialised in 87ms [ 0.147428] devtmpfs: initialized [ 0.148150] x86/mm: Memory block size: 128MB [ 0.150226] gcov: version magic: 0x41383552 [ 0.151516] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.152043] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.153168] pinctrl core: initialized pinctrl subsystem [ 0.154095] [ 0.154555] ************************************************************* [ 0.155006] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.156006] ** ** [ 0.157006] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.158005] ** ** [ 0.159005] ** This means that this kernel is built to expose internal ** [ 0.160006] ** IOMMU data structures, which may compromise security on ** [ 0.161027] ** your system. ** [ 0.162006] ** ** [ 0.163006] ** If you see this message and you are not debugging the ** [ 0.164008] ** kernel, report this immediately to your vendor! ** [ 0.165006] ** ** [ 0.166006] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.167008] ************************************************************* [ 0.168460] NET: Registered protocol family 16 [ 0.169336] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.170052] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.171028] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.172326] cpuidle: using governor menu [ 0.174642] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.176394] PCI: Using configuration type 1 for base access [ 0.178126] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.184068] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.185016] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.186103] cryptd: max_cpu_qlen set to 1000 [ 0.187242] ACPI: Added _OSI(Module Device) [ 0.189012] ACPI: Added _OSI(Processor Device) [ 0.190009] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.192008] ACPI: Added _OSI(Processor Aggregator Device) [ 0.198551] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.202422] ACPI: Interpreter enabled [ 0.203035] ACPI: PM: (supports S0 S3 S4 S5) [ 0.203971] ACPI: Using IOAPIC for interrupt routing [ 0.205050] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.207260] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.214109] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.216018] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.218008] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.220032] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.222937] acpiphp: Slot [2] registered [ 0.223069] acpiphp: Slot [5] registered [ 0.223913] acpiphp: Slot [6] registered [ 0.225049] acpiphp: Slot [3] registered [ 0.225901] acpiphp: Slot [4] registered [ 0.227055] acpiphp: Slot [7] registered [ 0.227863] acpiphp: Slot [8] registered [ 0.229042] acpiphp: Slot [9] registered [ 0.229847] acpiphp: Slot [10] registered [ 0.230042] acpiphp: Slot [11] registered [ 0.230853] acpiphp: Slot [12] registered [ 0.232053] acpiphp: Slot [13] registered [ 0.232887] acpiphp: Slot [14] registered [ 0.233064] acpiphp: Slot [15] registered [ 0.235048] acpiphp: Slot [16] registered [ 0.236056] acpiphp: Slot [17] registered [ 0.237073] acpiphp: Slot [18] registered [ 0.238044] acpiphp: Slot [19] registered [ 0.239050] acpiphp: Slot [20] registered [ 0.241053] acpiphp: Slot [21] registered [ 0.242056] acpiphp: Slot [22] registered [ 0.243042] acpiphp: Slot [23] registered [ 0.245120] acpiphp: Slot [24] registered [ 0.246087] acpiphp: Slot [25] registered [ 0.248092] acpiphp: Slot [26] registered [ 0.249054] acpiphp: Slot [27] registered [ 0.250068] acpiphp: Slot [28] registered [ 0.250945] acpiphp: Slot [29] registered [ 0.252081] acpiphp: Slot [30] registered [ 0.252966] acpiphp: Slot [31] registered [ 0.254045] PCI host bridge to bus 0000:00 [ 0.254953] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.256010] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.258008] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.259007] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.261009] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.263009] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.264128] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.266712] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.268910] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.273830] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.277557] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.279010] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.280023] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.284011] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.286689] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.289577] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.292047] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.294648] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.297013] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.305015] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.309006] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.312510] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.318020] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.324016] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.337014] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.345483] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.348622] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.354574] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.363023] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.370297] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.374353] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.376304] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.378338] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.381171] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.386027] iommu: Default domain type: Passthrough [ 0.387331] SCSI subsystem initialized [ 0.389120] ACPI: bus type USB registered [ 0.390099] usbcore: registered new interface driver usbfs [ 0.392032] usbcore: registered new interface driver hub [ 0.394056] usbcore: registered new device driver usb [ 0.396093] pps_core: LinuxPPS API ver. 1 registered [ 0.397006] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.400019] PTP clock support registered [ 0.402120] EDAC MC: Ver: 3.0.0 [ 0.403282] PCI: Using ACPI for IRQ routing [ 0.405610] NetLabel: Initializing [ 0.406006] NetLabel: domain hash size = 128 [ 0.406940] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.408074] NetLabel: unlabeled traffic allowed by default [ 0.409114] vgaarb: loaded [ 0.410297] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.412011] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.418343] clocksource: Switched to clocksource kvm-clock [ 0.501919] VFS: Disk quotas dquot_6.6.0 [ 0.502817] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.504320] *** VALIDATE ramfs *** [ 0.505064] *** VALIDATE hugetlbfs *** [ 0.506413] pnp: PnP ACPI init [ 0.507941] pnp: PnP ACPI: found 6 devices [ 0.520542] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.522670] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.524034] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.525395] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.526716] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.528340] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.530164] NET: Registered protocol family 2 [ 0.531988] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.535220] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.537236] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.540930] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.542990] TCP: Hash tables configured (established 65536 bind 65536) [ 0.544792] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.546680] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.548517] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.550730] NET: Registered protocol family 1 [ 0.552658] RPC: Registered named UNIX socket transport module. [ 0.554539] RPC: Registered udp transport module. [ 0.556366] RPC: Registered tcp transport module. [ 0.558175] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.560583] NET: Registered protocol family 44 [ 0.562232] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.564327] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.566395] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.568627] PCI: CLS 0 bytes, default 64 [ 0.570948] Unpacking initramfs... [ 1.866063] debug: unmapping init [mem 0xffff8e883cc64000-0xffff8e883ffcffff] [ 1.869583] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 1.871546] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 1.873894] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.347424] Initialise system trusted keyrings [ 2.350233] Key type blacklist registered [ 2.351667] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.365337] zbud: loaded [ 2.368752] *** VALIDATE nfs *** [ 2.370171] *** VALIDATE nfs4 *** [ 2.371915] pstore: using deflate compression [ 2.375066] Platform Keyring initialized [ 2.463812] NET: Registered protocol family 38 [ 2.465227] Key type asymmetric registered [ 2.466289] Asymmetric key parser 'x509' registered [ 2.467624] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.470134] io scheduler mq-deadline registered [ 2.471139] io scheduler kyber registered [ 2.472280] io scheduler bfq registered [ 2.473575] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.475721] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.477460] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.479539] ACPI: Power Button [PWRF] [ 2.484193] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.491050] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.500434] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.525830] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.552944] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.557174] Non-volatile memory driver v1.3 [ 2.558563] Linux agpgart interface v0.103 [ 2.584457] virtio_blk virtio1: [vda] 145176 512-byte logical blocks (74.3 MB/70.9 MiB) [ 2.587095] vda: detected capacity change from 0 to 74330112 [ 2.599183] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 2.601520] vdb: detected capacity change from 0 to 1073741824 [ 2.605845] libphy: Fixed MDIO Bus: probed [ 2.611870] usbcore: registered new interface driver usbserial_generic [ 2.614372] usbserial: USB Serial support registered for generic [ 2.615769] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 2.618871] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 2.620246] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 2.621956] mousedev: PS/2 mouse device common for all mice [ 2.624770] rtc_cmos 00:05: RTC can wake from S4 [ 2.627424] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 2.627660] rtc_cmos 00:05: registered as rtc0 [ 2.632375] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 2.632834] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 2.635022] intel_pstate: CPU model not supported [ 2.639635] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 2.644761] hid: raw HID events driver (C) Jiri Kosina [ 2.646459] usbcore: registered new interface driver usbhid [ 2.648104] usbhid: USB HID core driver [ 2.649415] drop_monitor: Initializing network drop monitor service [ 2.651724] Initializing XFRM netlink socket [ 2.653454] NET: Registered protocol family 10 [ 2.655838] Segment Routing with IPv6 [ 2.657100] NET: Registered protocol family 17 [ 2.658663] mpls_gso: MPLS GSO support [ 2.663417] RAS: Correctable Errors collector initialized. [ 2.665535] AVX version of gcm_enc/dec engaged. [ 2.666480] AES CTR mode by8 optimization enabled [ 2.725322] sched_clock: Marking stable (2725283886, 0)->(3460523446, -735239560) [ 2.727824] registered taskstats version 1 [ 2.729354] Loading compiled-in X.509 certificates [ 2.731432] zswap: loaded using pool lzo/zbud [ 2.751594] Key type big_key registered [ 2.764284] Key type encrypted registered [ 2.765824] ima: No TPM chip found, activating TPM-bypass! [ 2.767466] ima: Allocated hash algorithm: sha1 [ 2.768825] ima: No architecture policies found [ 2.770256] evm: Initialising EVM extended attributes: [ 2.772329] evm: security.selinux [ 2.773572] evm: security.ima [ 2.774483] evm: security.capability [ 2.775760] evm: HMAC attrs: 0x1 [ 2.778121] rtc_cmos 00:05: setting system clock to 2026-07-01 17:40:46 UTC (1782927646) [ 2.783401] debug: unmapping init [mem 0xffffffffb6203000-0xffffffffb63fffff] [ 2.786565] debug: unmapping init [mem 0xffffffffb4f82000-0xffffffffb5258fff] [ 2.797149] Write protecting the kernel read-only data: 28672k [ 2.800574] debug: unmapping init [mem 0xffffffffb3603000-0xffffffffb37fffff] [ 2.803657] debug: unmapping init [mem 0xffffffffb3f14000-0xffffffffb3ffffff] [ 2.830833] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 2.839104] systemd[1]: Detected virtualization kvm. [ 2.840497] systemd[1]: Detected architecture x86-64. [ 2.841978] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 2.864033] systemd[1]: No hostname configured. [ 2.865512] systemd[1]: Set hostname to . [ 2.866976] random: systemd: uninitialized urandom read (16 bytes read) [ 2.868335] systemd[1]: Initializing machine ID from random generator. [ 2.908359] random: ln: uninitialized urandom read (6 bytes read) [ 2.985190] random: systemd: uninitialized urandom read (16 bytes read) [ 2.987286] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 2.990561] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 2.994413] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Swap. [ OK ] Reached target Paths. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Timers. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Memstrack Anylazing Service. Starting Create Volatile Files and Directories... [ OK ] Listening on Journal Socket (/dev/log). Starting Setup Virtual Console... [ OK ] Reached target Sockets. Starting Apply Kernel Variables... Starting Journal Service... [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 3.580600] device-mapper: uevent: version 1.0.3 [ 3.582817] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 4.417560] random: fast init done [ 4.418128] virtio_net virtio0 ens2: renamed from eth0 [ 4.620446] scsi host0: ata_piix [ 4.648077] scsi host1: ata_piix [ 4.649415] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 4.652232] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 9.282480] random: crng init done [ 9.284379] random: 7 urandom warning(s) missed due to ratelimiting [ 9.482873] dracut-initqueue[599]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.212634] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.631106] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.952962] SELinux: Disabled at runtime. [ 12.012651] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.021264] systemd[1]: Detected virtualization kvm. [ 12.023161] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.661177] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.664922] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.671902] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.675922] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.681261] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.690440] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.700570] systemd[1]: proc-sys-fs-binfmt_misc.automount: Refusing to start, unit to trigger not loaded. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on udev Control Socket. [ OK ] Created slice User and Session Slice. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Slices. Starting Remount Root and Kernel File Systems... [ OK ] Created slice system-getty.slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. Mounting Kernel Debug File System... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. [ OK ] Reached target RPC Port Mapper. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... Mounting Huge Pages File System... Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice system-serial\x2dgetty.slice. Starting Apply Kernel Variables... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Started Disp[ 12.994763] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS atch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. Mounting POSIX Message Queue File System... [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted Huge Pages File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Mounted /mnt. [ 13.553842] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.996654] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 14.050801] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 14.168180] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 14.236155] EDAC sbridge: Ver: 1.1.2 [ 15.941988] Key type dns_resolver registered [ 16.312845] NFS: Registering the id_resolver key type [ 16.314851] Key type id_resolver registered [ 16.316700] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Rebuild Dynamic Linker Cache... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Login Service... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... [ OK ] Started Login Service. Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10[ 25.161423] hrtimer: interrupt took 3212639 ns (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg305-client login: [ 44.034700] libcfs: loading out-of-tree module taints kernel. [ 44.083624] Key type ._llcrypt registered [ 44.084960] Key type .llcrypt registered [ 44.342567] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 44.348405] alg: No test for adler32 (adler32-zlib) [ 45.331879] Lustre: Lustre: Build Version: 2.17.54_83_g6b808a3 [ 45.638092] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 47.248139] Key type lgssc registered [ 47.847168] Lustre: Echo OBD driver; http://www.lustre.org/ [ 91.392408] Lustre: Mounted lustre-client [ 93.662772] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 103.232054] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing check_logdir /tmp/testlogs/ [ 104.667229] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing yml_node [ 105.965932] Lustre: DEBUG MARKER: Client: 2.17.54.83 [ 106.726333] Lustre: DEBUG MARKER: MDS: 2.17.54.83 [ 107.480374] Lustre: DEBUG MARKER: OSS: 2.17.54.83 [ 107.966826] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Wed Jul 1 13:42:31 EDT 2026 [ 112.385106] Lustre: Unmounted lustre-client [ 123.385260] Key type lgssc unregistered [ 123.495450] LNet: 8399:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 123.497716] LNetError: 8399:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 123.509701] LNet: Removed LNI 192.168.203.5@tcp [ 123.777137] Key type .llcrypt unregistered [ 123.778848] Key type ._llcrypt unregistered [ 132.013273] Key type ._llcrypt registered [ 132.014636] Key type .llcrypt registered [ 132.029044] LNet: 8975:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 132.029044] mask usage: [+|-] ... [ 132.088140] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 132.094761] alg: No test for adler32 (adler32-zlib) [ 132.906996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 132.935071] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 132.944403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 134.250781] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 134.278448] LNet: 9273:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 134.281237] LNetError: 9273:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 134.289605] LNet: Removed LNI 192.168.203.5@tcp [ 134.486234] Key type .llcrypt unregistered [ 134.487802] Key type ._llcrypt unregistered [ 136.452330] Key type ._llcrypt registered [ 136.454876] Key type .llcrypt registered [ 136.942750] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 13:43:00 (1782927780) [ 137.004807] LNet: 9648:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 137.004807] mask usage: [+|-] ... [ 137.068036] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 137.075736] alg: No test for adler32 (adler32-zlib) [ 137.883673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 137.913704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 137.924407] LNet: 9689:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 137.933066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 137.955949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 137.964305] LNet: 9696:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 140.090757] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 13:43:03 (1782927783) [ 140.118474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 140.127993] LNet: 10268:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 140.140117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 140.174225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 142.302838] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 13:43:05 (1782927785) [ 142.338718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 142.347821] LNet: 10851:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 142.357495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 142.387811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 144.501133] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 13:43:07 (1782927787) [ 144.533044] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 144.544421] LNet: 11435:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 144.553948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 144.583832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 146.527575] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 13:43:09 (1782927789) [ 146.556939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 146.571026] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 146.601436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 146.766850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 148.824984] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 13:43:12 (1782927792) [ 148.854343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 148.863634] LNet: 12609:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 148.866781] LNet: 12609:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 148.876226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 148.902851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 149.036993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 149.186614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 151.211527] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 13:43:14 (1782927794) [ 151.237883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 151.258387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 151.288052] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 153.380075] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 13:43:16 (1782927796) [ 153.413947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 153.431263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 153.458423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 153.470191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 153.624922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 153.639605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 153.799523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 153.813169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 153.938600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 153.952333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 154.099064] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 154.114985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 154.263111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 154.277099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 154.431114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 154.444923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 154.581573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 154.594670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 154.734656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 154.750292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 154.884393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 154.897110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 155.022499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 155.033817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 155.172291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 155.186451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 155.321604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 155.335858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 157.547488] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 13:43:20 (1782927800) [ 157.577512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 157.587561] LNet: 14467:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 157.590119] LNet: 14467:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 157.599826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 157.631658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 157.645219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 159.739550] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 13:43:23 (1782927803) [ 159.788473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 159.805312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 159.832837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 159.844210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 161.931062] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 13:43:25 (1782927805) [ 161.965702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 161.983221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 162.014513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 162.029140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 164.083081] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 13:43:27 (1782927807) [ 164.123799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 164.142851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 164.176036] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 164.190459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 166.287621] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 13:43:29 (1782927809) [ 166.318558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 166.335547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 166.370520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 166.383889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 168.414406] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 13:43:31 (1782927811) [ 168.446384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 168.461557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 168.498108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 168.512989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 170.553827] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 13:43:33 (1782927813) [ 170.591629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 170.611585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 170.650943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 170.782160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 172.925111] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 13:43:36 (1782927816) [ 172.962516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 172.978566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 173.046128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 173.118339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 175.244336] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 13:43:38 (1782927818) [ 175.273073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 175.282795] LNet: 19193:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 175.286448] LNet: 19193:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 7 previous similar messages [ 175.296891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 175.342415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 175.411156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 177.482778] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 13:43:40 (1782927820) [ 177.515508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 177.533362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 177.709885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 177.800916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 180.108426] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 13:43:43 (1782927823) [ 180.140604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 180.157767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 180.193476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 180.212494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 182.379380] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 13:43:45 (1782927825) [ 182.425908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 182.445580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 182.472392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 182.491323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 184.686608] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 13:43:47 (1782927827) [ 184.727824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 184.744562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 184.774534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 186.848271] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 13:43:50 (1782927830) [ 186.911839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 186.927683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 186.954308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 189.025940] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 13:43:52 (1782927832) [ 189.054424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 189.072216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 189.109597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 189.251868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 191.233801] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 13:43:54 (1782927834) [ 191.259830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 191.275935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 191.308111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 191.319178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 193.426941] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 13:43:56 (1782927836) [ 193.455083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 193.468916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 193.497459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 193.511920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 195.676327] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 13:43:58 (1782927838) [ 195.705791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 195.722447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 195.756272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 195.770335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 197.952766] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 13:44:01 (1782927841) [ 197.987046] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 198.002076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 198.034711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 198.045297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 200.190120] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 13:44:03 (1782927843) [ 200.217546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 200.233308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 200.261205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 200.286982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 200.295183] LNetError: 25773:0:(peer.c:2234:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 200.312322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 200.326406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 202.336552] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 13:44:05 (1782927845) [ 202.375648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 202.394266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 202.424127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.XHp86QK4VA [ 202.447025] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.XHp86QK4VA [ 202.467505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.XHp86QK4VA [ 204.358806] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 13:44:07 (1782927847) [ 204.400794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 204.418247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 204.440427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 204.454493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 204.472097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 206.450342] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 13:44:09 (1782927849) [ 206.490509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 206.505466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 206.531571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 207.402937] LNet: Message forwarding will be enabled in 2s [ 210.464088] LNet: Message forwarding enabled [ 210.527978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 210.535946] LNet: Message forwarding will stop in 2s to 6s [ 214.682716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 215.128880] LNet: Message forwarding disabled [ 215.511349] LNet: Message forwarding will be enabled in 2s [ 217.601328] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 13:44:20 (1782927860) [ 217.628729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 217.638222] LNet: 28209:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 217.641149] LNet: 28209:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 14 previous similar messages [ 218.202727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 218.233355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 220.369594] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 13:44:23 (1782927863) [ 220.401802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 220.421571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 220.448601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 220.463717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 220.477852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 220.493854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 220.508682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 220.523650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 220.537919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 220.554371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 220.568488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 220.583303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 220.599480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 220.614765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 220.629423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 220.643236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 220.661067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 220.677385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 220.692459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 220.708172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 220.722846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 220.732808] LNetError: 28838:0:(router.c:624:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 222.787113] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 13:44:26 (1782927866) [ 222.826821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 222.847044] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 222.884607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 224.979671] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 13:44:28 (1782927868) [ 225.587789] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg305-server [ 226.152044] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 13:44:29 (1782927869) [ 226.180469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 226.198145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 226.222369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 226.231053] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 226.771729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.203.[6-10]@tcp [ 226.781513] LNet: 30552:0:(router.c:728:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 226.792318] LNetError: 30341:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.203.6@tcp is being used as a gateway but routing feature is not turned on [ 226.799669] LNetError: 30341:0:(lib-lnet.h:1266:lnet_set_route_aliveness()) route to tcp1 through 192.168.203.6@tcp has gone from up to down [ 227.810754] LNetError: 30341:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.203.6@tcp is being used as a gateway but routing feature is not turned on [ 227.815707] LNetError: 30341:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 1 previous similar message [ 228.837428] LNetError: 30341:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.203.6@tcp is being used as a gateway but routing feature is not turned on [ 228.955567] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 13:44:32 (1782927872) [ 229.547746] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg305-server [ 229.856292] LNetError: 30349:0:(lib-lnet.h:1266:lnet_set_route_aliveness()) route to tcp1 through 192.168.203.8@tcp has gone from up to down [ 230.058022] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 13:44:33 (1782927873) [ 230.087229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 230.096691] LNetError: 31481:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 230.105556] LNet: Removed LNI 192.168.203.5@tcp [ 230.116878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 230.138185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 230.147544] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 230.375220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.203.[6-11/2]@tcp [ 230.384036] LNet: 31603:0:(router.c:728:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 230.387946] LNet: 31603:0:(router.c:728:lnet_add_route()) Skipped 4 previous similar messages [ 230.396289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.203.[6-11/2]@tcp [ 232.449562] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 13:44:35 (1782927875) [ 232.481662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 233.504537] LNetError: 32188:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 233.512461] LNet: Removed LNI 192.168.203.5@tcp [ 233.519947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 233.550905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 233.568492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 233.579692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 233.601047] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 233.620537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 233.631719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 233.652345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 233.684366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 233.699918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 233.725863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 233.758597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 233.773550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 233.792188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 233.822378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 233.837533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 233.858947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 233.884563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 233.896440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 233.917618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 233.945765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 233.960392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 233.983700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 234.013161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 234.025896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 234.046960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 234.075730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 234.090486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 234.111098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 234.135849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 234.147402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 234.167390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 234.188652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 234.201767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 234.228727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 236.170290] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 13:44:39 (1782927879) [ 236.750568] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg305-server [ 237.268381] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 13:44:40 (1782927880) [ 237.816516] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg305-server [ 238.340789] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 13:44:41 (1782927881) [ 238.609350] Key type .llcrypt unregistered [ 238.610868] Key type ._llcrypt unregistered [ 240.548955] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 240.584152] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 240.586641] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 240.588750] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 242.793649] Key type ._llcrypt registered [ 242.795236] Key type .llcrypt registered [ 242.810426] LNet: 33872:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 242.810426] mask usage: [+|-] ... [ 242.870133] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 242.878455] alg: No test for adler32 (adler32-zlib) [ 243.682985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 243.706923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 243.720221] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 243.956163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 243.964848] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 244.658316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 244.665604] LNet: Removed LNI 10.1.2.3@tcp [ 244.705542] LNet: 34313:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 244.708501] LNetError: 34313:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 244.717505] LNet: Removed LNI 192.168.203.5@tcp [ 244.936319] Key type .llcrypt unregistered [ 244.937537] Key type ._llcrypt unregistered [ 248.511253] Key type ._llcrypt registered [ 248.512722] Key type .llcrypt registered [ 248.960579] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 13:44:52 (1782927892) [ 249.130262] Key type .llcrypt unregistered [ 249.131607] Key type ._llcrypt unregistered [ 251.094572] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 251.156967] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 251.158809] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 251.160918] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 253.350942] Key type ._llcrypt registered [ 253.352290] Key type .llcrypt registered [ 253.366478] LNet: 35346:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 253.366478] mask usage: [+|-] ... [ 253.423994] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 253.430568] alg: No test for adler32 (adler32-zlib) [ 254.236833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 254.260648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 254.272233] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 254.512259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 254.522606] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 255.381763] LNet: 35803:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 255.384237] LNetError: 35803:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 255.386568] LNet: Removed LNI 10.1.2.3@tcp [ 255.393329] LNet: Removed LNI 192.168.203.5@tcp [ 255.610305] Key type .llcrypt unregistered [ 255.611357] Key type ._llcrypt unregistered [ 259.313971] Key type ._llcrypt registered [ 259.315502] Key type .llcrypt registered [ 259.784451] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 13:45:02 (1782927902) [ 259.966387] Key type .llcrypt unregistered [ 259.968256] Key type ._llcrypt unregistered [ 261.933520] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 261.989474] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 261.991353] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 261.992732] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 264.208836] Key type ._llcrypt registered [ 264.209705] Key type .llcrypt registered [ 264.220359] LNet: 36837:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 264.220359] mask usage: [+|-] ... [ 264.263326] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 264.268337] alg: No test for adler32 (adler32-zlib) [ 265.069552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 265.090747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 265.102311] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 265.727402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 265.735804] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 266.376090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 266.386110] LNet: Removed LNI 10.1.2.3@tcp [ 266.392838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 266.409613] LNet: Removed LNI 10.1.2.31@tcp [ 266.415295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 266.424221] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 266.665656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 266.673377] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 266.990688] LNet: 37720:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 266.993827] LNetError: 37720:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg:0 not found [ 266.996573] LNet: Removed LNI 10.1.2.31@tcp [ 267.004424] LNet: Removed LNI 10.1.2.3@tcp [ 267.209352] Key type .llcrypt unregistered [ 267.211202] Key type ._llcrypt unregistered [ 270.503750] Key type ._llcrypt registered [ 270.504655] Key type .llcrypt registered [ 270.933863] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 13:45:14 (1782927914) [ 271.019154] LNet: 38510:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 271.019154] mask usage: [+|-] ... [ 271.066418] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 271.072281] alg: No test for adler32 (adler32-zlib) [ 271.872514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 271.890593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 271.898163] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 272.142903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 272.151794] LNet: 38671:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 272.154945] LNetError: 38671:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 272.162284] LNet: Removed LNI 192.168.203.5@tcp [ 272.168384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 272.188025] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.203.5@tcp [ 272.198618] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 272.432838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.203.5@tcp [ 272.450587] LNet: Removed LNI 192.168.203.5@tcp [ 272.696467] Key type .llcrypt unregistered [ 272.698221] Key type ._llcrypt unregistered [ 276.194889] Key type ._llcrypt registered [ 276.196407] Key type .llcrypt registered [ 276.625408] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 13:45:19 (1782927919) [ 277.211095] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg305-server [ 277.731555] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 13:45:21 (1782927921) [ 277.881174] Key type .llcrypt unregistered [ 277.882426] Key type ._llcrypt unregistered [ 279.641420] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 279.692265] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 279.693958] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 279.695923] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 281.904247] Key type ._llcrypt registered [ 281.905546] Key type .llcrypt registered [ 281.922269] LNet: 40179:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 281.922269] mask usage: [+|-] ... [ 281.970595] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 281.975879] alg: No test for adler32 (adler32-zlib) [ 282.775330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 282.796374] LNet: Added LNI 192.168.203.5@tcp1 [8/256/0/180] [ 283.040381] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 283.710947] LNet: 40638:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 283.714777] LNetError: 40638:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 283.718404] LNet: Removed LNI 192.168.203.5@tcp1 [ 283.727413] LNet: Removed LNI 10.1.2.3@tcp2 [ 283.945639] Key type .llcrypt unregistered [ 283.947224] Key type ._llcrypt unregistered [ 287.267796] Key type ._llcrypt registered [ 287.269458] Key type .llcrypt registered [ 287.695975] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 13:45:30 (1782927930) [ 287.778387] LNet: 41440:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 287.778387] mask usage: [+|-] ... [ 287.834091] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 287.840484] alg: No test for adler32 (adler32-zlib) [ 288.649480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 288.671301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 288.681517] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 288.958569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 288.965360] LNet: 41605:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 288.968183] LNetError: 41605:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 288.976287] LNet: Removed LNI 192.168.203.5@tcp [ 288.982368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 289.002815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 289.010984] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 291.157200] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 13:45:34 (1782927934) [ 291.196821] LNet: 42189:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 291.200377] LNetError: 42189:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 291.209713] LNet: Removed LNI 192.168.203.5@tcp [ 291.416349] Key type .llcrypt unregistered [ 291.418051] Key type ._llcrypt unregistered [ 293.177223] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 293.216096] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 293.218238] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 293.220187] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 295.407320] Key type ._llcrypt registered [ 295.409111] Key type .llcrypt registered [ 295.421562] LNet: 42471:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 295.421562] mask usage: [+|-] ... [ 295.474351] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 295.479263] alg: No test for adler32 (adler32-zlib) [ 296.275547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 296.293143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 296.303334] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 296.535480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 296.542614] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 297.190901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 297.199756] LNet: 42937:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 297.202167] LNetError: 42937:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 297.204665] LNet: Removed LNI 10.1.2.3@tcp [ 297.212245] LNet: Removed LNI 192.168.203.5@tcp [ 297.219646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 297.241171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 297.250188] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 297.252228] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 297.389605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 297.397158] LNet: Removed LNI 10.1.2.3@tcp [ 297.406211] LNet: Removed LNI 192.168.203.5@tcp [ 297.411744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 297.429275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 297.442055] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 297.444329] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 297.561977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 297.570164] LNet: Added LNI 192.168.203.5@tcp2 [8/256/0/180] [ 297.856860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 297.863986] LNet: 43127:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 297.865916] LNet: 43127:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 297.867974] LNetError: 43127:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 297.869796] LNetError: 43127:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 297.872061] LNet: Removed LNI 10.1.2.3@tcp [ 297.872919] LNet: Removed LNI 192.168.203.5@tcp [ 297.882226] LNet: Removed LNI 192.168.203.5@tcp2 [ 297.888316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 297.904836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 297.911561] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 297.913297] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 297.915544] LNet: Added LNI 192.168.203.5@tcp2 [8/256/0/180] [ 299.960275] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 13:45:43 (1782927943) [ 299.989606] LNet: 43716:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 299.993032] LNetError: 43716:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 299.995300] LNetError: 43716:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 299.997805] LNet: Removed LNI 10.1.2.3@tcp [ 299.998930] LNet: Removed LNI 192.168.203.5@tcp [ 300.004445] LNet: Removed LNI 192.168.203.5@tcp2 [ 300.234638] Key type .llcrypt unregistered [ 300.236559] Key type ._llcrypt unregistered [ 302.123079] Key type ._llcrypt registered [ 302.124527] Key type .llcrypt registered [ 302.139087] LNet: 43947:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 302.139087] mask usage: [+|-] ... [ 302.194959] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 302.201547] alg: No test for adler32 (adler32-zlib) [ 303.005281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 303.023678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 303.032832] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 303.282526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 303.288671] LNet: 44117:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 303.290776] LNetError: 44117:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 303.298254] LNet: Removed LNI 192.168.203.5@tcp [ 303.306322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 303.330889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 303.342092] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 303.524342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 303.541292] LNet: Removed LNI 192.168.203.5@tcp [ 303.548785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 303.570671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 303.581597] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 303.762866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 303.777366] LNet: Removed LNI 192.168.203.5@tcp [ 303.783365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 303.800719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 303.808765] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 303.938831] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 303.975938] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 303.977321] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 303.979058] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 306.114727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 306.122984] LNet: 44249:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 306.126679] LNet: 44249:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 306.129938] LNetError: 44249:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 306.132339] LNetError: 44249:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 306.140233] LNet: Removed LNI 192.168.203.5@tcp [ 306.146423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 306.166529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 306.176187] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 308.316030] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 13:45:51 (1782927951) [ 308.347267] LNet: 44840:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 308.349914] LNetError: 44840:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 308.359655] LNet: Removed LNI 192.168.203.5@tcp [ 308.582621] Key type .llcrypt unregistered [ 308.584149] Key type ._llcrypt unregistered [ 310.351047] Key type ._llcrypt registered [ 310.352377] Key type .llcrypt registered [ 310.364685] LNet: 45070:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 310.364685] mask usage: [+|-] ... [ 310.415575] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 310.420671] alg: No test for adler32 (adler32-zlib) [ 311.226686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 311.246994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 311.255023] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 311.535159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 311.541394] LNet: 45243:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 311.543282] LNetError: 45243:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 311.550257] LNet: Removed LNI 192.168.203.5@tcp [ 311.556882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 311.578437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 311.587473] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 313.531994] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 13:45:56 (1782927956) [ 313.570803] LNet: 45830:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 313.573448] LNetError: 45830:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 313.583111] LNet: Removed LNI 192.168.203.5@tcp [ 313.789194] Key type .llcrypt unregistered [ 313.790151] Key type ._llcrypt unregistered [ 315.571676] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 315.625856] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 315.630345] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 315.632904] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 317.804699] Key type ._llcrypt registered [ 317.805558] Key type .llcrypt registered [ 317.817563] LNet: 46110:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 317.817563] mask usage: [+|-] ... [ 317.869711] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 317.875609] alg: No test for adler32 (adler32-zlib) [ 318.669864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 318.688099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 318.698525] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 318.978591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 318.987097] LNet: 46283:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 318.989471] LNetError: 46283:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 318.998285] LNet: Removed LNI 192.168.203.5@tcp [ 319.004595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 319.025126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 319.034166] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 321.133940] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 13:46:04 (1782927964) [ 321.164054] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 321.172731] LNet: 46876:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 321.177735] LNetError: 46876:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 321.183727] LNet: Removed LNI 192.168.203.5@tcp [ 321.192493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 321.229587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 323.179084] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 13:46:06 (1782927966) [ 323.209979] LNet: 47454:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 323.427409] Key type .llcrypt unregistered [ 323.428693] Key type ._llcrypt unregistered [ 325.344631] Key type ._llcrypt registered [ 325.346207] Key type .llcrypt registered [ 325.362331] LNet: 47686:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 325.362331] mask usage: [+|-] ... [ 325.419899] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 325.426058] alg: No test for adler32 (adler32-zlib) [ 326.222757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 326.243581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 326.252364] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 326.514423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 326.520414] LNet: 47858:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 326.522367] LNetError: 47858:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 326.531278] LNet: Removed LNI 192.168.203.5@tcp [ 326.536724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 326.552664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 326.559494] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 328.453715] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 13:46:11 (1782927971) [ 328.482030] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 328.490775] LNet: 48446:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 328.492825] LNetError: 48446:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 328.497424] LNet: Removed LNI 192.168.203.5@tcp [ 328.505333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 328.531445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 330.301865] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 13:46:13 (1782927973) [ 330.740278] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 331.208689] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 13:46:14 (1782927974) [ 331.647147] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 332.128376] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 13:46:15 (1782927975) [ 332.584376] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 333.058415] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 13:46:16 (1782927976) [ 333.483349] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 333.915893] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 13:46:17 (1782927977) [ 334.403178] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 334.956739] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing ========================================================== 13:46:18 (1782927978) [ 334.986850] LNet: 50493:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 335.241599] Key type .llcrypt unregistered [ 335.242825] Key type ._llcrypt unregistered [ 337.041824] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 337.071872] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 337.074088] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 337.076239] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 339.266722] Key type ._llcrypt registered [ 339.267829] Key type .llcrypt registered [ 339.279855] LNet: 50782:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 339.279855] mask usage: [+|-] ... [ 339.333305] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 339.341314] alg: No test for adler32 (adler32-zlib) [ 340.135220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 340.155075] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 340.163786] LNet: Added LNI 192.168.203.5@tcp [8/512/0/180] [ 340.165711] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 340.167431] LNet: 50820:0:(api-ni.c:2914:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 340.200200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 340.206923] LNet: 50840:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 340.209242] LNetError: 50840:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 340.211413] LNet: Removed LNI 10.1.2.3@tcp [ 340.213073] LNetError: 50840:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 340.221243] LNet: Removed LNI 192.168.203.5@tcp [ 340.229137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 340.252500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 340.259164] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 340.280196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 340.286978] LNet: 50864:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 340.289208] LNetError: 50864:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 340.298274] LNet: Removed LNI 192.168.203.5@tcp [ 340.305352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 340.334149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 340.343338] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 340.344979] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 340.346616] LNet: 50872:0:(api-ni.c:2914:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 340.379734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 340.388031] LNet: 50890:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 340.390889] LNetError: 50890:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 340.393646] LNet: Removed LNI 10.1.2.3@tcp [ 340.395852] LNetError: 50890:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 340.410273] LNet: Removed LNI 192.168.203.5@tcp [ 340.417789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 340.441067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 340.449239] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 340.451668] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 340.452784] LNet: 50898:0:(api-ni.c:2914:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 340.512771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 340.519462] LNet: 50925:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 340.521731] LNetError: 50925:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 340.523727] LNet: Removed LNI 10.1.2.3@tcp [ 340.524621] LNetError: 50925:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 340.532265] LNet: Removed LNI 192.168.203.5@tcp [ 340.538865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 340.560678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 340.569562] LNet: Added LNI 192.168.203.5@tcp [8/256/0/182] [ 340.571533] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 340.601452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 340.607339] LNet: 50953:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 340.609259] LNetError: 50953:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 340.611025] LNet: Removed LNI 10.1.2.3@tcp [ 340.611834] LNetError: 50953:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 340.621272] LNet: Removed LNI 192.168.203.5@tcp [ 340.626762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 340.646965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 340.654054] LNet: Added LNI 192.168.203.5@tcp [8/256/0/182] [ 340.656101] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 340.657133] LNet: 50961:0:(api-ni.c:2914:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 340.682208] LNet: Removed LNI 10.1.2.3@tcp [ 340.693320] LNet: Removed LNI 192.168.203.5@tcp [ 340.904278] Key type .llcrypt unregistered [ 340.905134] Key type ._llcrypt unregistered [ 344.343652] Key type ._llcrypt registered [ 344.345273] Key type .llcrypt registered [ 344.785731] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 13:46:28 (1782927988) [ 344.871325] LNet: 51774:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 344.871325] mask usage: [+|-] ... [ 344.919565] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 344.925163] alg: No test for adler32 (adler32-zlib) [ 345.730695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 345.755319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 345.767187] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 346.051403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.203.5 [ 346.062119] LNet: 51936:0:(config.c:1627:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 346.067476] LNet: Removed LNI 192.168.203.5@tcp [ 348.109643] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 13:46:31 (1782927991) [ 348.146357] LNet: 52498:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 348.150715] LNetError: 52498:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 348.161150] LNet: Removed LNI 192.168.203.5@tcp [ 348.400294] Key type .llcrypt unregistered [ 348.401792] Key type ._llcrypt unregistered [ 350.575083] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 350.681486] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 352.958490] Key type ._llcrypt registered [ 352.960358] Key type .llcrypt registered [ 352.976185] LNet: 52787:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 352.976185] mask usage: [+|-] ... [ 353.025600] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 353.031670] alg: No test for adler32 (adler32-zlib) [ 354.023819] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 356.531456] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 13:46:39 (1782927999) [ 356.576528] LNet: 53411:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 356.581579] LNetError: 53411:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 356.592528] LNet: Removed LNI 10.1.2.3@tcp [ 356.828292] Key type .llcrypt unregistered [ 356.829727] Key type ._llcrypt unregistered [ 358.885125] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 358.979088] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 358.981576] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 358.984201] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 361.262727] Key type ._llcrypt registered [ 361.264196] Key type .llcrypt registered [ 361.279958] LNet: 53690:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 361.279958] mask usage: [+|-] ... [ 361.328317] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 361.334659] alg: No test for adler32 (adler32-zlib) [ 362.152533] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 364.275458] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 13:46:47 (1782928007) [ 364.761263] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 365.316764] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 13:46:48 (1782928008) [ 365.348289] LNet: 54593:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 365.351443] LNetError: 54593:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 365.358521] LNet: Removed LNI 10.1.2.3@tcp [ 365.592584] Key type .llcrypt unregistered [ 365.594209] Key type ._llcrypt unregistered [ 367.535640] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 367.637605] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 367.639720] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 367.641910] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 369.897891] Key type ._llcrypt registered [ 369.899347] Key type .llcrypt registered [ 369.910759] LNet: 54874:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 369.910759] mask usage: [+|-] ... [ 369.957163] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 369.962707] alg: No test for adler32 (adler32-zlib) [ 370.760992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 370.788770] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 373.444382] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 13:46:56 (1782928016) [ 373.562423] LNet: 55829:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 373.564859] LNetError: 55829:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 373.571477] LNet: Removed LNI 10.1.2.3@tcp [ 373.798357] Key type .llcrypt unregistered [ 373.799494] Key type ._llcrypt unregistered [ 375.605387] Key type ._llcrypt registered [ 375.606912] Key type .llcrypt registered [ 375.622511] LNet: 56050:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 375.622511] mask usage: [+|-] ... [ 375.678301] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 375.684598] alg: No test for adler32 (adler32-zlib) [ 376.486711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 376.495695] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 379.337547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.105@tcp [ 379.545655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp -d 192.168.203.105@tcp -m GET -e local_interrupt -r 1 [ 379.560479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp -d 192.168.203.5@tcp -m GET -e local_interrupt -r 1 [ 379.573305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.105@tcp [ 379.581324] LNet: There was an unexpected network error while writing to 192.168.203.105: rc = -22 [ 379.651177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 379.661377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 379.673721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 379.685261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 379.698351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 379.709139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 379.720254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 379.731559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 379.737597] LNet: 56263:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 379.745784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 379.761095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 379.775927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 379.790413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 379.805528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 379.819134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 379.833366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 379.989100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp -d 192.168.203.105@tcp -m GET -e local_dropped -r 1 [ 380.002187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp -d 192.168.203.5@tcp -m GET -e local_dropped -r 1 [ 380.017018] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.105@tcp [ 380.088730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 380.098629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 380.108534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 380.120440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 380.134706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 380.149106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 380.163790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 380.178810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 380.189365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 380.202926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 380.218077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 380.232048] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 380.244281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 380.259152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 380.272799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 380.430386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp -d 192.168.203.105@tcp -m GET -e local_aborted -r 1 [ 380.444297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp -d 192.168.203.5@tcp -m GET -e local_aborted -r 1 [ 380.457275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.105@tcp [ 380.464354] LNet: There was an unexpected network error while writing to 192.168.203.105: rc = -22 [ 380.466394] LNet: Skipped 1 previous similar message [ 380.532446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 380.543790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 380.554432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 380.565429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 380.575925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 380.576177] LNet: 1 local NIs in recovery (showing 1): 192.168.203.5@tcp [ 380.587835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 380.598497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 380.610523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 380.619020] LNet: 56482:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 380.623211] LNet: 56482:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 380.632301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 380.646202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 380.657304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 380.668031] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 380.679124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 380.693968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 380.706377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 380.850378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp -d 192.168.203.105@tcp -m GET -e local_no_route -r 1 [ 380.860992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp -d 192.168.203.5@tcp -m GET -e local_no_route -r 1 [ 380.873071] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.105@tcp [ 380.938761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 380.949939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 380.962078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 380.973796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 380.985753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 380.997944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 381.007746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 381.018196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 381.030300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 381.044446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 381.054416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 381.064173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 381.076419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 381.086612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 381.097778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 381.246501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp -d 192.168.203.105@tcp -m GET -e local_timeout -r 1 [ 381.259453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp -d 192.168.203.5@tcp -m GET -e local_timeout -r 1 [ 381.276292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.105@tcp [ 381.355591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 381.368912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 381.380967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 381.393086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 381.404242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 381.415861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 381.427058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 381.437157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 381.446367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 381.456506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 381.465811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 381.475145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 381.484880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 381.494911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 381.505814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 381.659167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp -d 192.168.203.105@tcp -m GET -e local_error -r 1 [ 381.670192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp -d 192.168.203.5@tcp -m GET -e local_error -r 1 [ 381.681810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.105@tcp [ 381.688750] LNet: There was an unexpected network error while writing to 192.168.203.105: rc = -22 [ 381.692485] LNet: Skipped 3 previous similar messages [ 381.761608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 381.774130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 381.784809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 381.793218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 381.802338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 381.811257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 381.820279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 381.830507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 381.837287] LNet: 56809:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 381.839650] LNet: 56809:0:(api-ni.c:294:recovery_interval_set()) Skipped 2 previous similar messages [ 381.849098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 381.864737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 381.877712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 381.887960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 381.899149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 381.909617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 381.920071] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 382.008304] LNet: 56850:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 382.011262] LNetError: 56850:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 382.024261] LNet: Removed LNI 192.168.203.5@tcp [ 382.151447] Key type .llcrypt unregistered [ 382.152698] Key type ._llcrypt unregistered [ 387.285522] Key type ._llcrypt registered [ 387.286473] Key type .llcrypt registered [ 387.709902] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 13:47:10 (1782928030) [ 387.949272] Key type .llcrypt unregistered [ 387.950119] Key type ._llcrypt unregistered [ 389.850371] Key type ._llcrypt registered [ 389.852209] Key type .llcrypt registered [ 389.865815] LNet: 57884:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 389.865815] mask usage: [+|-] ... [ 389.922394] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 389.928941] alg: No test for adler32 (adler32-zlib) [ 390.734944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 390.744097] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 393.584802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.105@tcp [ 396.699493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 396.710894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 396.718036] LNet: Added LNI 192.168.203.5@tcp1 [8/256/0/180] [ 397.181356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp -d 192.168.203.105@tcp -m GET -e local_interrupt -r 1 [ 397.197498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp -d 192.168.203.105@tcp1 -m GET -e local_interrupt -r 1 [ 397.215608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp -d 192.168.203.5@tcp -m GET -e local_interrupt -r 1 [ 397.232217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp -d 192.168.203.5@tcp1 -m GET -e local_interrupt -r 1 [ 397.250102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp1 -d 192.168.203.105@tcp -m GET -e local_interrupt -r 1 [ 397.265752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp1 -d 192.168.203.105@tcp1 -m GET -e local_interrupt -r 1 [ 397.284189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp1 -d 192.168.203.5@tcp -m GET -e local_interrupt -r 1 [ 397.301890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp1 -d 192.168.203.5@tcp1 -m GET -e local_interrupt -r 1 [ 397.319984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.105@tcp [ 397.331659] LNet: There was an unexpected network error while writing to 192.168.203.105: rc = -22 [ 397.334660] LNet: 1 local NIs in recovery (showing 1): 192.168.203.5@tcp [ 397.410552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 397.422163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 397.436464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 397.450472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 397.463400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 397.475439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 397.488769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 397.503266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 397.512385] LNet: 58311:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 397.521479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 397.536651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 397.548145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 397.561264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 397.575434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 397.592499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 397.610071] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 397.772995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp -d 192.168.203.105@tcp -m GET -e local_dropped -r 1 [ 397.789417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp -d 192.168.203.105@tcp1 -m GET -e local_dropped -r 1 [ 397.806511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp -d 192.168.203.5@tcp -m GET -e local_dropped -r 1 [ 397.820286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp -d 192.168.203.5@tcp1 -m GET -e local_dropped -r 1 [ 397.835419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp1 -d 192.168.203.105@tcp -m GET -e local_dropped -r 1 [ 397.848853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp1 -d 192.168.203.105@tcp1 -m GET -e local_dropped -r 1 [ 397.861830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp1 -d 192.168.203.5@tcp -m GET -e local_dropped -r 1 [ 397.873080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp1 -d 192.168.203.5@tcp1 -m GET -e local_dropped -r 1 [ 397.886442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.105@tcp [ 397.894427] LNet: There was an unexpected network error while writing to 192.168.203.105: rc = -22 [ 397.897351] LNet: Skipped 2 previous similar messages [ 397.967291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 397.981680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 397.995581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 398.008632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 398.019985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 398.031414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 398.042384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 398.054992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 398.064590] LNet: 58432:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 398.073470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 398.086290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 398.101356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 398.112272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 398.126528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 398.142814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 398.155957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 398.332287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp -d 192.168.203.105@tcp -m GET -e local_aborted -r 1 [ 398.347360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp -d 192.168.203.105@tcp1 -m GET -e local_aborted -r 1 [ 398.361232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp -d 192.168.203.5@tcp -m GET -e local_aborted -r 1 [ 398.378598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp -d 192.168.203.5@tcp1 -m GET -e local_aborted -r 1 [ 398.395440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp1 -d 192.168.203.105@tcp -m GET -e local_aborted -r 1 [ 398.409582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp1 -d 192.168.203.105@tcp1 -m GET -e local_aborted -r 1 [ 398.426647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp1 -d 192.168.203.5@tcp -m GET -e local_aborted -r 1 [ 398.442397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp1 -d 192.168.203.5@tcp1 -m GET -e local_aborted -r 1 [ 398.458467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.105@tcp [ 398.536242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 398.546969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 398.559840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 398.571329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 398.585583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 398.598655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 398.611537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 398.624314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 398.635477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 398.645267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 398.654742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 398.666085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 398.680362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 398.692586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 398.707640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 398.889954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp -d 192.168.203.105@tcp -m GET -e local_no_route -r 1 [ 398.909326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp -d 192.168.203.105@tcp1 -m GET -e local_no_route -r 1 [ 398.926147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp -d 192.168.203.5@tcp -m GET -e local_no_route -r 1 [ 398.943160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp -d 192.168.203.5@tcp1 -m GET -e local_no_route -r 1 [ 398.958959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp1 -d 192.168.203.105@tcp -m GET -e local_no_route -r 1 [ 398.974641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp1 -d 192.168.203.105@tcp1 -m GET -e local_no_route -r 1 [ 398.988561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp1 -d 192.168.203.5@tcp -m GET -e local_no_route -r 1 [ 399.000262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp1 -d 192.168.203.5@tcp1 -m GET -e local_no_route -r 1 [ 399.011286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.105@tcp [ 399.017487] LNet: There was an unexpected network error while writing to 192.168.203.105: rc = -22 [ 399.019441] LNet: Skipped 6 previous similar messages [ 399.094433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 399.104953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 399.120879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 399.136227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 399.148205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 399.162225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 399.178701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 399.194491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 399.205750] LNet: 58674:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 399.209410] LNet: 58674:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 399.219464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 399.235352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 399.249867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 399.263974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 399.279383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 399.295524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 399.310405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 399.506600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp -d 192.168.203.105@tcp -m GET -e local_timeout -r 1 [ 399.523054] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp -d 192.168.203.105@tcp1 -m GET -e local_timeout -r 1 [ 399.540422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp -d 192.168.203.5@tcp -m GET -e local_timeout -r 1 [ 399.558930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp -d 192.168.203.5@tcp1 -m GET -e local_timeout -r 1 [ 399.576523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp1 -d 192.168.203.105@tcp -m GET -e local_timeout -r 1 [ 399.598774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp1 -d 192.168.203.105@tcp1 -m GET -e local_timeout -r 1 [ 399.621344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp1 -d 192.168.203.5@tcp -m GET -e local_timeout -r 1 [ 399.640992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp1 -d 192.168.203.5@tcp1 -m GET -e local_timeout -r 1 [ 399.660510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.105@tcp [ 399.746779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 399.760681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 399.774961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 399.788610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 399.802828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 399.819046] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 399.835127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 399.849517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 399.865080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 399.879327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 399.893675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 399.906656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 399.920823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 399.937625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 399.952980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 400.135372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp -d 192.168.203.105@tcp -m GET -e local_error -r 1 [ 400.151612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp -d 192.168.203.105@tcp1 -m GET -e local_error -r 1 [ 400.169034] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp -d 192.168.203.5@tcp -m GET -e local_error -r 1 [ 400.186369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp -d 192.168.203.5@tcp1 -m GET -e local_error -r 1 [ 400.202367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp1 -d 192.168.203.105@tcp -m GET -e local_error -r 1 [ 400.218123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp1 -d 192.168.203.105@tcp1 -m GET -e local_error -r 1 [ 400.233919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp1 -d 192.168.203.5@tcp -m GET -e local_error -r 1 [ 400.254882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp1 -d 192.168.203.5@tcp1 -m GET -e local_error -r 1 [ 400.272018] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.105@tcp [ 400.354742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 400.368933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 400.383991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 400.396585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 400.409096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 400.425142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 400.441631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 400.457808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 400.470884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 400.483515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 400.497839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 400.512808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 400.526456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 400.541415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 400.556911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 402.761946] LNet: 58992:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 402.764462] LNetError: 58992:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 402.768516] LNet: Removed LNI 192.168.203.5@tcp [ 403.814274] LNet: Removed LNI 192.168.203.5@tcp1 [ 403.935524] Key type .llcrypt unregistered [ 403.936930] Key type ._llcrypt unregistered [ 409.045397] Key type ._llcrypt registered [ 409.046970] Key type .llcrypt registered [ 409.493963] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 13:47:32 (1782928052) [ 409.756227] Key type .llcrypt unregistered [ 409.757543] Key type ._llcrypt unregistered [ 411.759869] Key type ._llcrypt registered [ 411.761414] Key type .llcrypt registered [ 411.773480] LNet: 60024:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 411.773480] mask usage: [+|-] ... [ 411.823230] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 411.830052] alg: No test for adler32 (adler32-zlib) [ 412.633144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 412.643990] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 415.452845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.105@tcp [ 415.686111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp -d 192.168.203.105@tcp -m GET -e remote_dropped -r 1 [ 415.699969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp -d 192.168.203.5@tcp -m GET -e remote_dropped -r 1 [ 415.715917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.105@tcp [ 415.724653] LNet: There was an unexpected network error while writing to 192.168.203.105: rc = -22 [ 415.799941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 415.812827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 415.828040] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 415.839576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 415.853399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 415.869200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 415.883121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 415.895288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 415.902458] LNet: 60238:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 415.910635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 415.922265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 415.937960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 415.952827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 415.964396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 415.975195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 415.988412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 416.165024] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp -d 192.168.203.105@tcp -m GET -e remote_error -r 1 [ 416.180737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp -d 192.168.203.5@tcp -m GET -e remote_error -r 1 [ 416.199492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.105@tcp [ 416.291138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 416.305538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 416.320045] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 416.332434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 416.345665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 416.360273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 416.374474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 416.387288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 416.400893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 416.414925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 416.428792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 416.441883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 416.454583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 416.466863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 416.479603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 416.654821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp -d 192.168.203.105@tcp -m GET -e remote_timeout -r 1 [ 416.671740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp -d 192.168.203.5@tcp -m GET -e remote_timeout -r 1 [ 416.689505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.105@tcp [ 416.701676] LNet: There was an unexpected network error while writing to 192.168.203.105: rc = -22 [ 416.704836] LNet: Skipped 1 previous similar message [ 416.777161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 416.791192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 416.803518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 416.816953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 416.829931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 416.842477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 416.855231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 416.869793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 416.878200] LNet: 60456:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 416.881238] LNet: 60456:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 416.890237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 416.905396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 416.919662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 416.933476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 416.948636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 416.963705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 416.978763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 417.075146] LNet: 60497:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 417.077316] LNetError: 60497:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 417.088396] LNet: Removed LNI 192.168.203.5@tcp [ 417.225318] Key type .llcrypt unregistered [ 417.226704] Key type ._llcrypt unregistered [ 422.203425] Key type ._llcrypt registered [ 422.204261] Key type .llcrypt registered [ 422.646653] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 13:47:45 (1782928065) [ 422.909215] Key type .llcrypt unregistered [ 422.910670] Key type ._llcrypt unregistered [ 424.928156] Key type ._llcrypt registered [ 424.929444] Key type .llcrypt registered [ 424.942462] LNet: 61531:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 424.942462] mask usage: [+|-] ... [ 424.995025] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 425.000992] alg: No test for adler32 (adler32-zlib) [ 425.802137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 425.812951] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 428.589897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.105@tcp [ 430.960903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 430.973536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 430.980423] LNet: Added LNI 192.168.203.5@tcp1 [8/256/0/180] [ 431.380375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp -d 192.168.203.105@tcp -m GET -e remote_dropped -r 1 [ 431.390929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp -d 192.168.203.105@tcp1 -m GET -e remote_dropped -r 1 [ 431.403650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp -d 192.168.203.5@tcp -m GET -e remote_dropped -r 1 [ 431.415569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp -d 192.168.203.5@tcp1 -m GET -e remote_dropped -r 1 [ 431.428169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp1 -d 192.168.203.105@tcp -m GET -e remote_dropped -r 1 [ 431.439632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp1 -d 192.168.203.105@tcp1 -m GET -e remote_dropped -r 1 [ 431.451340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp1 -d 192.168.203.5@tcp -m GET -e remote_dropped -r 1 [ 431.468172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp1 -d 192.168.203.5@tcp1 -m GET -e remote_dropped -r 1 [ 431.481271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.105@tcp [ 431.487971] LNet: There was an unexpected network error while writing to 192.168.203.105: rc = -22 [ 431.490601] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.105@tcp [ 431.556618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 431.568090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 431.580032] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 431.590591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 431.600404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 431.614225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 431.625207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 431.638320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 431.644922] LNet: 61958:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 431.652071] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 431.663015] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 431.672881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 431.683259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 431.696808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 431.710951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 431.721813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 431.731940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 431.882568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp -d 192.168.203.105@tcp -m GET -e remote_error -r 1 [ 431.896663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp -d 192.168.203.105@tcp1 -m GET -e remote_error -r 1 [ 431.909151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp -d 192.168.203.5@tcp -m GET -e remote_error -r 1 [ 431.921655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp -d 192.168.203.5@tcp1 -m GET -e remote_error -r 1 [ 431.932420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp1 -d 192.168.203.105@tcp -m GET -e remote_error -r 1 [ 431.943149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp1 -d 192.168.203.105@tcp1 -m GET -e remote_error -r 1 [ 431.954715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp1 -d 192.168.203.5@tcp -m GET -e remote_error -r 1 [ 431.965521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp1 -d 192.168.203.5@tcp1 -m GET -e remote_error -r 1 [ 431.977341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.105@tcp [ 432.047170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 432.059118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 432.070647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 432.079449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 432.087927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 432.097454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 432.107678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 432.118825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 432.129852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 432.142186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 432.152892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 432.164410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 432.175361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 432.189301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 432.202942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 432.216655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 432.383511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp -d 192.168.203.105@tcp -m GET -e remote_timeout -r 1 [ 432.395978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp -d 192.168.203.105@tcp1 -m GET -e remote_timeout -r 1 [ 432.409732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp -d 192.168.203.5@tcp -m GET -e remote_timeout -r 1 [ 432.423148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp -d 192.168.203.5@tcp1 -m GET -e remote_timeout -r 1 [ 432.434688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp1 -d 192.168.203.105@tcp -m GET -e remote_timeout -r 1 [ 432.448706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp1 -d 192.168.203.105@tcp1 -m GET -e remote_timeout -r 1 [ 432.461832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp1 -d 192.168.203.5@tcp -m GET -e remote_timeout -r 1 [ 432.476805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp1 -d 192.168.203.5@tcp1 -m GET -e remote_timeout -r 1 [ 432.490606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.105@tcp [ 432.497625] LNet: There was an unexpected network error while writing to 192.168.203.105: rc = -22 [ 432.499719] LNet: Skipped 3 previous similar messages [ 432.567470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 432.578489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 432.589137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 432.599105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 432.610462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 432.622600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 432.634554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 432.646247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 432.652963] LNet: 62204:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 432.654730] LNet: 62204:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 432.660187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 432.672679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 432.682634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 432.692842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 432.703306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 432.712968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 432.723278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 432.734421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 434.832398] LNet: 62282:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 434.835849] LNetError: 62282:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 434.839376] LNet: Removed LNI 192.168.203.5@tcp [ 434.851327] LNet: Removed LNI 192.168.203.5@tcp1 [ 435.000255] Key type .llcrypt unregistered [ 435.001221] Key type ._llcrypt unregistered [ 439.914692] Key type ._llcrypt registered [ 439.916103] Key type .llcrypt registered [ 440.364452] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 13:48:03 (1782928083) [ 440.871688] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 441.389379] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 13:48:04 (1782928084) [ 441.619365] Key type .llcrypt unregistered [ 441.620317] Key type ._llcrypt unregistered [ 443.417268] Key type ._llcrypt registered [ 443.418368] Key type .llcrypt registered [ 443.433078] LNet: 63607:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 443.433078] mask usage: [+|-] ... [ 443.495892] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 443.502894] alg: No test for adler32 (adler32-zlib) [ 444.296350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 444.303584] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 447.154025] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.105@tcp [ 447.381923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp -d 192.168.203.105@tcp -m GET -e network_timeout -r 1 [ 447.399954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp -d 192.168.203.5@tcp -m GET -e network_timeout -r 1 [ 447.418303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.105@tcp [ 447.426950] LNet: There was an unexpected network error while writing to 192.168.203.105: rc = -22 [ 447.505262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 447.519358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 447.535419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 447.550492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 447.564389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 447.578218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 447.589899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 447.602390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 447.609270] LNet: 63821:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 447.617176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 447.629470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 447.643517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 447.657185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 447.671105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 447.685364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 447.777240] LNet: 63860:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 447.779418] LNetError: 63860:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 447.790291] LNet: Removed LNI 192.168.203.5@tcp [ 447.952193] Key type .llcrypt unregistered [ 447.953330] Key type ._llcrypt unregistered [ 453.538244] Key type ._llcrypt registered [ 453.539128] Key type .llcrypt registered [ 453.550140] LNet: 64308:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 453.550140] mask usage: [+|-] ... [ 453.597357] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 453.601740] alg: No test for adler32 (adler32-zlib) [ 454.395115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 454.403927] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 457.104705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.105@tcp [ 459.696745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 459.707651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 459.714609] LNet: Added LNI 192.168.203.5@tcp1 [8/256/0/180] [ 460.163686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp -d 192.168.203.105@tcp -m GET -e network_timeout -r 1 [ 460.179854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp -d 192.168.203.105@tcp1 -m GET -e network_timeout -r 1 [ 460.196759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp -d 192.168.203.5@tcp -m GET -e network_timeout -r 1 [ 460.213583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp -d 192.168.203.5@tcp1 -m GET -e network_timeout -r 1 [ 460.230093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp1 -d 192.168.203.105@tcp -m GET -e network_timeout -r 1 [ 460.244918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp1 -d 192.168.203.105@tcp1 -m GET -e network_timeout -r 1 [ 460.259491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp1 -d 192.168.203.5@tcp -m GET -e network_timeout -r 1 [ 460.276601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp1 -d 192.168.203.5@tcp1 -m GET -e network_timeout -r 1 [ 460.292470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.105@tcp [ 460.301836] LNet: There was an unexpected network error while writing to 192.168.203.105: rc = -22 [ 460.374342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 460.387242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 460.399733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 460.415420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 460.426985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 460.437318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 460.447797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 460.459935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 460.466856] LNet: 64736:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 460.475347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 460.488599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 460.501720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 460.516861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 460.531641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 460.546495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 462.573930] LNet: There was an unexpected network error while writing to 192.168.203.105: rc = -22 [ 462.624072] LNet: 1 local NIs in recovery (showing 1): 192.168.203.5@tcp [ 462.666887] LNet: 64809:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 462.670620] LNetError: 64809:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 462.674567] LNet: Removed LNI 192.168.203.5@tcp [ 462.683278] LNet: Removed LNI 192.168.203.5@tcp1 [ 462.815447] Key type .llcrypt unregistered [ 462.816520] Key type ._llcrypt unregistered [ 467.594727] Key type ._llcrypt registered [ 467.596279] Key type .llcrypt registered [ 468.009870] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 13:48:31 (1782928111) [ 468.072401] LNet: 65607:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 468.072401] mask usage: [+|-] ... [ 468.118246] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 468.123068] alg: No test for adler32 (adler32-zlib) [ 468.914130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 468.928590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 468.934846] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 469.133046] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 469.141847] LNet: Added LNI 192.168.203.5@tcp1 [8/256/0/180] [ 469.394461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.5@tcp [ 469.414383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 469.430027] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 469.440987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 469.452366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 469.462163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 469.472409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.203.5@tcp [ 469.478522] LNetError: 65906:0:(api-ni.c:4400:lnet_ni_set_healthv()) manually adding local NI 192.168.203.5@tcp to recovery [ 469.984263] LNet: 1 local NIs in recovery (showing 1): 192.168.203.5@tcp [ 469.984305] LNet: There was an unexpected network error while writing to 192.168.203.5: rc = -22 [ 471.008292] LNet: There was an unexpected network error while writing to 192.168.203.5: rc = -22 [ 471.597865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 475.104302] LNet: There was an unexpected network error while writing to 192.168.203.5: rc = -22 [ 475.788663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 475.801491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 475.825535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 475.834841] LNet: 66014:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 475.838476] LNetError: 66014:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 475.841816] LNet: Removed LNI 192.168.203.5@tcp [ 475.849287] LNet: Removed LNI 192.168.203.5@tcp1 [ 475.856858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 475.878586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 475.889921] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 476.145528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 476.155885] LNet: Added LNI 192.168.203.5@tcp1 [8/256/0/180] [ 476.452731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.5@tcp [ 476.468660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 476.495209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 476.516286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 476.530237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 476.544330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 476.559335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 476.572551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.203.5@tcp [ 476.581969] LNetError: 66286:0:(api-ni.c:4400:lnet_ni_set_healthv()) manually adding local NI 192.168.203.5@tcp to recovery [ 478.944283] LNet: There was an unexpected network error while writing to 192.168.203.5: rc = -22 [ 478.947090] LNet: Skipped 1 previous similar message [ 479.741329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 486.112474] LNet: There was an unexpected network error while writing to 192.168.203.5: rc = -22 [ 486.116183] LNet: Skipped 1 previous similar message [ 487.043013] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 487.054831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 487.066359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 489.344270] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 13:48:52 (1782928132) [ 489.398397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 489.407878] LNet: 67002:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 489.410145] LNetError: 67002:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 489.412969] LNetError: 67002:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 489.415903] LNet: Removed LNI 192.168.203.5@tcp [ 489.421542] LNet: Removed LNI 192.168.203.5@tcp1 [ 489.431257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 489.450141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 489.458972] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 489.698313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 489.706333] LNet: Added LNI 192.168.203.5@tcp1 [8/256/0/180] [ 489.999662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.5@tcp [ 490.023905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 490.034462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 490.046808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 490.058223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 490.069651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 490.080670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.5@tcp --health 0 [ 490.114823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 490.464130] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.5@tcp [ 497.632306] LNet: There was an unexpected network error while writing to 192.168.203.5: rc = -22 [ 497.635427] LNet: Skipped 2 previous similar messages [ 506.561592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 506.621617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 506.638308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 506.652229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.5@tcp --health 500 [ 506.687146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 513.910733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 513.936482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 513.944472] LNet: 67518:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 513.947243] LNetError: 67518:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 513.949821] LNetError: 67518:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 513.952507] LNet: Removed LNI 192.168.203.5@tcp [ 513.957256] LNet: Removed LNI 192.168.203.5@tcp1 [ 513.964134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 513.982950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 513.993095] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 514.238350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 514.249749] LNet: Added LNI 192.168.203.5@tcp1 [8/256/0/180] [ 514.539184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.5@tcp [ 514.552996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 514.576852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 514.593762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 514.609746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 514.624983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 514.641578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 514.659271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.5@tcp --health 0 [ 516.000266] LNet: There was an unexpected network error while writing to 192.168.203.5: rc = -22 [ 516.004419] LNet: Skipped 1 previous similar message [ 516.794017] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 526.146770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 526.160781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 526.174261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 528.400443] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 13:49:31 (1782928171) [ 528.533158] LNet: 68574:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 528.535272] LNetError: 68574:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 528.538602] LNetError: 68574:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 528.540746] LNet: Removed LNI 192.168.203.5@tcp [ 528.556339] LNet: Removed LNI 192.168.203.5@tcp1 [ 528.785602] Key type .llcrypt unregistered [ 528.786742] Key type ._llcrypt unregistered [ 530.839671] Key type ._llcrypt registered [ 530.840765] Key type .llcrypt registered [ 530.853716] LNet: 68796:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 530.853716] mask usage: [+|-] ... [ 530.905329] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 530.910403] alg: No test for adler32 (adler32-zlib) [ 531.715396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 531.726565] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 535.207660] Lustre: DEBUG MARKER: Initial discovery [ 535.239776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.105@tcp [ 536.053519] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 536.071735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 536.091591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.105@tcp [ 536.102638] LNet: There was an unexpected network error while writing to 192.168.203.105: rc = -22 [ 536.113103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 536.131850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.203.105@tcp -r 1 -m GET -l 3 [ 536.147740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.203.105@tcp -r 1 -m GET -e local_error [ 536.166087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.203.105@tcp -r 1 -m PUT -l 6 [ 536.629317] Lustre: DEBUG MARKER: Force oleg305-client.virtnet to discover 192.168.203.105@tcp (in background) [ 536.669741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.105@tcp [ 536.864104] LNet: 1 local NIs in recovery (showing 1): 192.168.203.5@tcp [ 537.161494] Lustre: DEBUG MARKER: Force oleg305-server to discover 192.168.203.5@tcp [ 538.015579] Lustre: DEBUG MARKER: Wait for 69331 [ 539.744182] LNet: There was an unexpected network error while writing to 192.168.203.105: rc = -22 [ 540.234796] Lustre: DEBUG MARKER: Finished wait on 69331 [ 540.261297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay del -a [ 540.276566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 540.373851] LNet: 69782:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 540.377839] LNetError: 69782:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 540.386305] LNet: Removed LNI 192.168.203.5@tcp [ 540.535499] Key type .llcrypt unregistered [ 540.536983] Key type ._llcrypt unregistered [ 545.872354] Key type ._llcrypt registered [ 545.873606] Key type .llcrypt registered [ 546.348981] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 13:49:49 (1782928189) [ 546.525703] Key type .llcrypt unregistered [ 546.527603] Key type ._llcrypt unregistered [ 548.620555] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 548.661579] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 548.664515] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 548.667524] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 550.870757] Key type ._llcrypt registered [ 550.871827] Key type .llcrypt registered [ 550.883715] LNet: 70823:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 550.883715] mask usage: [+|-] ... [ 550.938074] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 550.943928] alg: No test for adler32 (adler32-zlib) [ 551.748670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 551.770270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 551.780428] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 552.038931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 552.047954] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 554.851347] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 13:49:58 (1782928198) [ 554.882179] LNet: 71848:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 554.884362] LNetError: 71848:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 554.886519] LNet: Removed LNI 10.1.2.3@tcp [ 554.895713] LNet: Removed LNI 192.168.203.5@tcp [ 555.156411] Key type .llcrypt unregistered [ 555.157597] Key type ._llcrypt unregistered [ 557.298335] Key type ._llcrypt registered [ 557.300378] Key type .llcrypt registered [ 557.314736] LNet: 72104:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 557.314736] mask usage: [+|-] ... [ 557.373092] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 557.379257] alg: No test for adler32 (adler32-zlib) [ 558.184187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 558.207545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 558.218844] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 558.464381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 558.471521] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 558.807841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.203.5@tcp [ 559.840139] LNetError: Unexpected error -22 connecting to 192.168.203.5@tcp at host 192.168.203.5:988 [ 560.850633] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 13:50:04 (1782928204) [ 560.880109] LNet: 72978:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 560.882644] LNetError: 72978:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 560.885272] LNet: Removed LNI 10.1.2.3@tcp [ 560.890686] LNet: Removed LNI 192.168.203.5@tcp [ 561.120963] Key type .llcrypt unregistered [ 561.122879] Key type ._llcrypt unregistered [ 563.183770] Key type ._llcrypt registered [ 563.184893] Key type .llcrypt registered [ 563.196898] LNet: 73217:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 563.196898] mask usage: [+|-] ... [ 563.253610] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 563.259862] alg: No test for adler32 (adler32-zlib) [ 564.066447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 564.089567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 564.101602] LNet: Added LNI 192.168.203.5@tcp1 [8/256/0/180] [ 564.375506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 564.383579] LNet: Added LNI 192.168.203.5@tcp2 [8/256/0/180] [ 564.716582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.203.5@tcp1 --nid 192.168.203.5@tcp2 [ 565.892866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.5@tcp1 192.168.203.5@tcp2 [ 565.899031] LNetError: 73820:0:(lib-move.c:2007:lnet_handle_find_routed_path()) Unable to send message from 192.168.203.5@tcp1 to 192.168.203.5@tcp2 - Route table may be misconfigured [ 565.905190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.5@tcp2 192.168.203.5@tcp1 [ 567.736365] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 13:50:11 (1782928211) [ 567.766301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 567.775939] LNet: 74386:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 567.777933] LNetError: 74386:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 567.780537] LNet: Removed LNI 192.168.203.5@tcp1 [ 568.806476] LNet: Removed LNI 192.168.203.5@tcp2 [ 568.813293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 568.831035] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 568.838660] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 569.061599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 569.070209] LNet: Added LNI 192.168.203.5@tcp1 [8/256/0/180] [ 569.331491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.5@tcp [ 569.344721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.5@tcp 192.168.203.5@tcp [ 569.358178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.5@tcp1 192.168.203.5@tcp1 [ 569.374459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.203.5@tcp -d 192.168.203.5@tcp -e network_timeout [ 569.388914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.203.5@tcp -d 192.168.203.5@tcp1 -e network_timeout [ 569.403745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.203.5@tcp1 -d 192.168.203.5@tcp -e network_timeout [ 569.413955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.203.5@tcp1 -d 192.168.203.5@tcp1 -e network_timeout [ 569.425045] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.5@tcp [ 569.431035] LNet: There was an unexpected network error while writing to 192.168.203.5: rc = -22 [ 569.439389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 569.471327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 569.505609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 569.824064] LNet: 1 local NIs in recovery (showing 1): 192.168.203.5@tcp [ 571.458662] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 13:50:14 (1782928214) [ 571.493572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 571.502415] LNet: 75249:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 571.505327] LNetError: 75249:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 571.508251] LNetError: 75249:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 571.510074] LNet: Removed LNI 192.168.203.5@tcp [ 572.518334] LNet: Removed LNI 192.168.203.5@tcp1 [ 572.524137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 572.547444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 572.614514] LNet: 75290:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 572.743235] Key type .llcrypt unregistered [ 572.744448] Key type ._llcrypt unregistered [ 575.993623] Key type ._llcrypt registered [ 575.994831] Key type .llcrypt registered [ 576.391924] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 13:50:19 (1782928219) [ 576.465193] LNet: 76060:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 576.465193] mask usage: [+|-] ... [ 576.503829] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 576.507608] alg: No test for adler32 (adler32-zlib) [ 577.311959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 577.769907] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 578.279651] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 13:50:21 (1782928221) [ 578.318579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 578.326980] LNet: 76384:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 578.338044] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 578.359717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 578.368031] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 578.596532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 578.605541] LNet: Added LNI 192.168.203.5@tcp1 [8/256/0/180] [ 578.889434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.5@tcp [ 578.902323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.5@tcp1 [ 578.916889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.5@tcp1 [ 580.871729] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 13:50:24 (1782928224) [ 581.427292] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg305-server [ 581.949893] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 13:50:25 (1782928225) [ 582.509387] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg305-server [ 583.019519] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 13:50:26 (1782928226) [ 583.584897] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg305-server [ 584.130961] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 13:50:27 (1782928227) [ 584.679078] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg305-server [ 585.193848] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 13:50:28 (1782928228) [ 585.771032] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg305-server [ 586.279393] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 13:50:29 (1782928229) [ 586.857133] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg305-server [ 587.419501] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 13:50:30 (1782928230) [ 588.001126] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg305-server [ 588.556351] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 13:50:31 (1782928231) [ 589.134047] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg305-server [ 589.724511] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 13:50:32 (1782928232) [ 590.376329] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg305-server [ 590.976900] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 13:50:34 (1782928234) [ 591.017595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 591.030709] LNet: 80255:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 591.035166] LNetError: 80255:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 591.039481] LNet: Removed LNI 192.168.203.5@tcp [ 592.102671] LNet: Removed LNI 192.168.203.5@tcp1 [ 592.110802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 592.128980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 592.137209] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 592.393542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 592.425445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.5@tcp [ 592.485881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 592.493373] LNet: 80405:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 592.496549] LNetError: 80405:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 592.499339] LNetError: 80405:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 592.506316] LNet: Removed LNI 192.168.203.5@tcp [ 592.514159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 592.532713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 592.540926] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 592.793534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.5@tcp [ 592.849646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 593.894273] LNet: Removed LNI 192.168.203.5@tcp [ 593.901308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 593.920840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 593.931720] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 594.168642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 594.201036] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.5@tcp [ 594.264337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 594.271730] LNetError: 80697:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 594.273834] LNetError: 80697:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 594.280337] LNet: Removed LNI 192.168.203.5@tcp [ 594.287781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 594.304341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 594.312647] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 594.578831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.5@tcp [ 594.640911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 594.649980] LNet: 80842:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 594.653703] LNet: 80842:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 595.686276] LNet: Removed LNI 192.168.203.5@tcp [ 595.693661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 595.712722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 595.724345] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 595.945413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 595.977438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.5@tcp [ 596.048397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 597.094283] LNet: Removed LNI 192.168.203.5@tcp [ 597.100742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 597.117579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 597.124759] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 597.364443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.5@tcp [ 597.426551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 597.433564] LNetError: 81136:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 597.436286] LNetError: 81136:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 597.443351] LNet: Removed LNI 192.168.203.5@tcp [ 597.451290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 597.468959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 597.476932] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 597.710971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 597.740828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.5@tcp [ 598.852151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 598.859304] LNet: 81292:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 598.861856] LNet: 81292:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 599.910261] LNet: Removed LNI 192.168.203.5@tcp [ 599.917665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 599.937135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 599.946462] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 600.192064] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.5@tcp [ 600.261955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 601.318270] LNet: Removed LNI 192.168.203.5@tcp [ 601.326377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 601.349255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 601.359761] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 601.609715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 601.639968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.5@tcp [ 602.761241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 602.769416] LNetError: 81594:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 602.772522] LNetError: 81594:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 602.781435] LNet: Removed LNI 192.168.203.5@tcp [ 602.789317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 602.811784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 602.822799] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 603.093783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.5@tcp [ 603.176248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 604.198490] LNet: Removed LNI 192.168.203.5@tcp [ 604.207682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 604.229531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 604.239847] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 604.510566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 604.537495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.5@tcp [ 604.612430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 605.670292] LNet: Removed LNI 192.168.203.5@tcp [ 605.678720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 605.697905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 605.706703] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 605.950305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.5@tcp [ 606.035715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 607.078414] LNet: Removed LNI 192.168.203.5@tcp [ 607.087521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 607.111436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 607.123854] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 607.393284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 607.413536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 607.421167] LNet: 82163:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 607.424099] LNet: 82163:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 5 previous similar messages [ 607.434531] LNet: Removed LNI 192.168.203.5@tcp [ 607.443823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 607.466375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 607.476384] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 607.735268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 610.019349] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 13:50:53 (1782928253) [ 610.060198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 610.078179] LNet: Removed LNI 192.168.203.5@tcp [ 610.090480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 610.113846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 610.125156] LNet: Added LNI 192.168.203.5@tcp231 [8/256/0/180] [ 610.588779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 610.605334] LNet: Removed LNI 192.168.203.5@tcp231 [ 610.614151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 610.639648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 610.648338] LNet: Added LNI 192.168.203.5@tcp231 [8/256/0/180] [ 610.650767] LNet: 83013:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 610.828733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 610.845545] LNet: Removed LNI 192.168.203.5@tcp231 [ 610.852035] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 610.863815] LNet: Added LNI 192.168.203.5@tcp231 [8/256/0/180] [ 611.309387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 611.318562] LNetError: 83157:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 611.321824] LNetError: 83157:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 6 previous similar messages [ 611.329391] LNet: Removed LNI 192.168.203.5@tcp231 [ 611.337608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 611.365017] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 611.373627] LNet: Added LNI 192.168.203.5@tcp231 [8/256/0/0] [ 611.375460] LNet: 83165:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 611.506648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 611.525307] LNet: Removed LNI 192.168.203.5@tcp231 [ 611.531637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 611.540862] LNet: Added LNI 192.168.203.5@tcp231 [8/256/0/0] [ 611.937344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 611.953685] LNet: Removed LNI 192.168.203.5@tcp231 [ 611.962335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 611.988381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 611.998210] LNet: Added LNI 192.168.203.5@tcp231 [8/256/0/60] [ 612.151928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 612.166539] LNet: Removed LNI 192.168.203.5@tcp231 [ 612.173574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 612.185565] LNet: Added LNI 192.168.203.5@tcp231 [8/256/0/60] [ 612.568239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 612.585342] LNet: Removed LNI 192.168.203.5@tcp231 [ 612.593401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 612.620791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 612.632133] LNet: Added LNI 192.168.203.5@tcp231 [8/256/0/180] [ 612.634872] LNet: 83467:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 612.637914] LNet: 83467:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 612.766965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 612.783681] LNet: Removed LNI 192.168.203.5@tcp231 [ 612.790107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 612.801374] LNet: Added LNI 192.168.203.5@tcp231 [8/256/0/180] [ 615.347647] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 13:50:58 (1782928258) [ 615.387676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 615.407577] LNet: Removed LNI 192.168.203.5@tcp231 [ 615.416771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 615.440431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 615.448664] LNet: Added LNI 192.168.203.5@tcp232 [8/256/0/180] [ 615.709435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 615.723781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 615.740536] LNet: Removed LNI 192.168.203.5@tcp232 [ 615.746791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 615.756190] LNet: Added LNI 192.168.203.5@tcp232 [8/256/0/180] [ 615.992246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 616.005146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 616.020468] LNet: Removed LNI 192.168.203.5@tcp232 [ 616.027993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 616.035926] LNet: Added LNI 192.168.203.5@tcp232 [8/256/0/180] [ 616.246715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 616.258643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 616.273485] LNet: Removed LNI 192.168.203.5@tcp232 [ 616.278896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 616.286203] LNet: Added LNI 192.168.203.5@tcp232 [8/256/0/180] [ 616.512822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 616.526302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 616.547485] LNet: Removed LNI 192.168.203.5@tcp232 [ 616.553341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 618.325327] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 13:51:01 (1782928261) [ 618.632244] Key type .llcrypt unregistered [ 618.633088] Key type ._llcrypt unregistered [ 620.516488] Key type ._llcrypt registered [ 620.517860] Key type .llcrypt registered [ 620.536578] LNet: 85531:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 620.536578] mask usage: [+|-] ... [ 620.603127] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 620.609116] alg: No test for adler32 (adler32-zlib) [ 621.417663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 621.429705] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 624.266799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.105@tcp [ 627.057516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 627.071486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 627.083049] LNet: Added LNI 192.168.203.5@tcp1 [8/256/0/180] [ 627.612286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp -d 192.168.203.105@tcp -m GET -e local_interrupt -r 2 [ 627.629702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp -d 192.168.203.105@tcp1 -m GET -e local_interrupt -r 2 [ 627.645275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp -d 192.168.203.5@tcp -m GET -e local_interrupt -r 2 [ 627.662869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp -d 192.168.203.5@tcp1 -m GET -e local_interrupt -r 2 [ 627.677797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp1 -d 192.168.203.105@tcp -m GET -e local_interrupt -r 2 [ 627.694744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp1 -d 192.168.203.105@tcp1 -m GET -e local_interrupt -r 2 [ 627.713163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp1 -d 192.168.203.5@tcp -m GET -e local_interrupt -r 2 [ 627.729557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp1 -d 192.168.203.5@tcp1 -m GET -e local_interrupt -r 2 [ 627.744720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.5@tcp 192.168.203.105@tcp [ 627.752295] LNet: There was an unexpected network error while writing to 192.168.203.105: rc = -22 [ 627.755558] LNet: 1 local NIs in recovery (showing 1): 192.168.203.5@tcp [ 627.849730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 627.860280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 627.871667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 627.886151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 627.896883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 627.909270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 627.923862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 627.940608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 627.950494] LNet: 85962:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 627.957428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 627.971668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 627.983713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 627.996681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 628.008293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 628.022328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 628.217924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp -d 192.168.203.105@tcp -m GET -e local_dropped -r 2 [ 628.233256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp -d 192.168.203.105@tcp1 -m GET -e local_dropped -r 2 [ 628.249261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp -d 192.168.203.5@tcp -m GET -e local_dropped -r 2 [ 628.264447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp -d 192.168.203.5@tcp1 -m GET -e local_dropped -r 2 [ 628.279615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp1 -d 192.168.203.105@tcp -m GET -e local_dropped -r 2 [ 628.294543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp1 -d 192.168.203.105@tcp1 -m GET -e local_dropped -r 2 [ 628.307262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp1 -d 192.168.203.5@tcp -m GET -e local_dropped -r 2 [ 628.319719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp1 -d 192.168.203.5@tcp1 -m GET -e local_dropped -r 2 [ 628.331848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.5@tcp 192.168.203.105@tcp [ 628.339154] LNet: There was an unexpected network error while writing to 192.168.203.105: rc = -22 [ 628.439849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 628.450519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 628.461297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 628.472447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 628.484789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 628.498424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 628.510655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 628.522516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 628.530020] LNet: 86082:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 628.538066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 628.553548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 628.569211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 628.584624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 628.597576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 628.610480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 628.812390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp -d 192.168.203.105@tcp -m GET -e local_aborted -r 2 [ 628.825342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp -d 192.168.203.105@tcp1 -m GET -e local_aborted -r 2 [ 628.839883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp -d 192.168.203.5@tcp -m GET -e local_aborted -r 2 [ 628.853780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp -d 192.168.203.5@tcp1 -m GET -e local_aborted -r 2 [ 628.868184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp1 -d 192.168.203.105@tcp -m GET -e local_aborted -r 2 [ 628.882751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp1 -d 192.168.203.105@tcp1 -m GET -e local_aborted -r 2 [ 628.897898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp1 -d 192.168.203.5@tcp -m GET -e local_aborted -r 2 [ 628.915296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp1 -d 192.168.203.5@tcp1 -m GET -e local_aborted -r 2 [ 628.931125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.5@tcp 192.168.203.105@tcp [ 629.041486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 629.056437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 629.069873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 629.081916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 629.094562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 629.106862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 629.119925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 629.132023] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 629.143377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 629.155663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 629.167826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 629.178754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 629.190516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 629.202128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 629.397672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp -d 192.168.203.105@tcp -m GET -e local_no_route -r 2 [ 629.412675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp -d 192.168.203.105@tcp1 -m GET -e local_no_route -r 2 [ 629.428080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp -d 192.168.203.5@tcp -m GET -e local_no_route -r 2 [ 629.442067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp -d 192.168.203.5@tcp1 -m GET -e local_no_route -r 2 [ 629.458207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp1 -d 192.168.203.105@tcp -m GET -e local_no_route -r 2 [ 629.471553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp1 -d 192.168.203.105@tcp1 -m GET -e local_no_route -r 2 [ 629.484549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp1 -d 192.168.203.5@tcp -m GET -e local_no_route -r 2 [ 629.498630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp1 -d 192.168.203.5@tcp1 -m GET -e local_no_route -r 2 [ 629.513207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.5@tcp 192.168.203.105@tcp [ 629.522758] LNet: There was an unexpected network error while writing to 192.168.203.105: rc = -22 [ 629.524974] LNet: Skipped 2 previous similar messages [ 629.621029] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 629.637145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 629.651183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 629.664413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 629.679530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 629.691716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 629.705297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 629.717560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 629.724633] LNet: 86323:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 629.727051] LNet: 86323:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 629.733802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 629.745316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 629.756918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 629.767497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 629.780181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 629.792220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 630.004883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp -d 192.168.203.105@tcp -m GET -e local_timeout -r 2 [ 630.019626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp -d 192.168.203.105@tcp1 -m GET -e local_timeout -r 2 [ 630.035213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp -d 192.168.203.5@tcp -m GET -e local_timeout -r 2 [ 630.048071] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp -d 192.168.203.5@tcp1 -m GET -e local_timeout -r 2 [ 630.063822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp1 -d 192.168.203.105@tcp -m GET -e local_timeout -r 2 [ 630.077114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp1 -d 192.168.203.105@tcp1 -m GET -e local_timeout -r 2 [ 630.090588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp1 -d 192.168.203.5@tcp -m GET -e local_timeout -r 2 [ 630.104040] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp1 -d 192.168.203.5@tcp1 -m GET -e local_timeout -r 2 [ 630.118881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.5@tcp 192.168.203.105@tcp [ 630.220378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 630.231671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 630.242808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 630.254119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 630.265390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 630.276957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 630.288534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 630.299575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 630.311184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 630.323041] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 630.334991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 630.346173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 630.357191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 630.367898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 630.552408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp -d 192.168.203.105@tcp -m GET -e remote_dropped -r 2 [ 630.568257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp -d 192.168.203.105@tcp1 -m GET -e remote_dropped -r 2 [ 630.585929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp -d 192.168.203.5@tcp -m GET -e remote_dropped -r 2 [ 630.603317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp -d 192.168.203.5@tcp1 -m GET -e remote_dropped -r 2 [ 630.616502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp1 -d 192.168.203.105@tcp -m GET -e remote_dropped -r 2 [ 630.630199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp1 -d 192.168.203.105@tcp1 -m GET -e remote_dropped -r 2 [ 630.643661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp1 -d 192.168.203.5@tcp -m GET -e remote_dropped -r 2 [ 630.657638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp1 -d 192.168.203.5@tcp1 -m GET -e remote_dropped -r 2 [ 630.671537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.5@tcp 192.168.203.105@tcp [ 630.680752] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.105@tcp [ 630.771442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 630.784662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 630.797763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 630.809498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 630.821447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 630.838120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 630.853263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 630.867127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 630.878905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 630.893022] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 630.904863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 630.917134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 630.928551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 630.939961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 633.166714] LNet: 86636:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 633.169177] LNetError: 86636:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 633.172060] LNet: Removed LNI 192.168.203.5@tcp [ 633.180422] LNet: Removed LNI 192.168.203.5@tcp1 [ 633.320589] Key type .llcrypt unregistered [ 633.321598] Key type ._llcrypt unregistered [ 638.836116] Key type ._llcrypt registered [ 638.837131] Key type .llcrypt registered [ 639.262331] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 13:51:22 (1782928282) [ 639.330239] LNet: 87440:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 639.330239] mask usage: [+|-] ... [ 639.380507] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 639.385611] alg: No test for adler32 (adler32-zlib) [ 640.182858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 640.199810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 640.209133] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 640.498377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 640.509351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.203.5@tcp [ 640.517342] LNetError: 87613:0:(api-ni.c:4400:lnet_ni_set_healthv()) manually adding local NI 192.168.203.5@tcp to recovery [ 640.545786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.203.5@tcp [ 640.576799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 642.612644] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 13:51:25 (1782928285) [ 642.642958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 642.654490] LNet: 88196:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 642.656864] LNetError: 88196:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 642.664668] LNet: Removed LNI 192.168.203.5@tcp [ 642.673647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 642.695579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 642.703734] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 642.956483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 642.965912] LNet: Added LNI 192.168.203.5@tcp1 [8/256/0/180] [ 643.286060] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.203.5@tcp [ 643.293942] LNetError: 88467:0:(api-ni.c:4400:lnet_ni_set_healthv()) manually adding local NI 192.168.203.5@tcp to recovery [ 643.744196] LNet: 1 local NIs in recovery (showing 1): 192.168.203.5@tcp [ 643.744669] LNet: There was an unexpected network error while writing to 192.168.203.5: rc = -22 [ 644.364169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.203.5@tcp [ 648.429937] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 13:51:31 (1782928291) [ 649.020188] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg305-server [ 649.549196] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 13:51:32 (1782928292) [ 650.139918] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg305-server [ 650.651311] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 13:51:33 (1782928293) [ 651.197758] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg305-server [ 651.752754] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 13:51:35 (1782928295) [ 651.794429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 651.808077] LNet: 90096:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 651.812418] LNetError: 90096:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 651.816465] LNet: Removed LNI 192.168.203.5@tcp [ 651.823499] LNet: Removed LNI 192.168.203.5@tcp1 [ 651.830859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 651.855022] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 651.867389] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 652.109218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 652.132048] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 652.149079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 652.174269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 652.190773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 652.206581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 654.190474] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 13:51:37 (1782928297) [ 654.234190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 654.245498] LNet: 90822:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 654.250083] LNetError: 90822:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 654.252627] LNetError: 90822:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 654.261504] LNet: Removed LNI 192.168.203.5@tcp [ 654.268888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 654.288971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 654.296958] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 656.461588] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 13:51:39 (1782928299) [ 656.501221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 656.512953] LNet: 91523:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 656.516940] LNetError: 91523:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 656.525590] LNet: Removed LNI 192.168.203.5@tcp [ 656.532983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 656.552899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 656.562556] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 658.777504] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 13:51:42 (1782928302) [ 659.254562] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 659.818868] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 13:51:43 (1782928303) [ 659.973652] LNet: Removed LNI 192.168.203.5@tcp [ 660.198458] Key type .llcrypt unregistered [ 660.199560] Key type ._llcrypt unregistered [ 662.203802] Key type ._llcrypt registered [ 662.204940] Key type .llcrypt registered [ 662.217919] LNet: 92786:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 662.217919] mask usage: [+|-] ... [ 662.269178] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 662.274495] alg: No test for adler32 (adler32-zlib) [ 663.073992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 663.082857] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 665.844152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.105@tcp [ 667.256473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.203.105@tcp [ 667.368140] LNet: 92973:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 667.371445] LNetError: 92973:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 667.383329] LNet: Removed LNI 192.168.203.5@tcp [ 667.532257] Key type .llcrypt unregistered [ 667.533702] Key type ._llcrypt unregistered [ 671.228301] Key type ._llcrypt registered [ 671.230039] Key type .llcrypt registered [ 671.657745] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 672.173923] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 672.699324] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 13:51:55 (1782928315) [ 673.186152] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 [ 673.726908] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 13:51:57 (1782928317) [ 674.291503] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg305-server [ 674.813810] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 13:51:58 (1782928318) [ 675.374469] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg305-server [ 675.877641] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 13:51:59 (1782928319) [ 676.171426] LNet: 95016:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 676.171426] mask usage: [+|-] ... [ 676.296466] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 676.350299] alg: No test for adler32 (adler32-zlib) [ 677.214550] Lustre: Lustre: Build Version: 2.17.54_83_g6b808a3 [ 677.307861] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 678.888177] Key type lgssc registered [ 679.287554] Lustre: Echo OBD driver; http://www.lustre.org/ [ 687.192066] Key type lgssc unregistered [ 687.306348] LNet: 96143:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 687.309457] LNetError: 96143:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 687.319276] LNet: Removed LNI 192.168.203.5@tcp [ 687.564415] Key type .llcrypt unregistered [ 687.565546] Key type ._llcrypt unregistered [ 692.465619] Key type ._llcrypt registered [ 692.466623] Key type .llcrypt registered [ 692.914478] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 13:52:16 (1782928336) [ 692.985046] LNet: 96938:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 692.985046] mask usage: [+|-] ... [ 693.034673] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 693.041715] alg: No test for adler32 (adler32-zlib) [ 693.841334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 693.863037] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 693.877491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 693.893496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 693.911862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 695.866203] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 13:52:19 (1782928339) [ 695.901591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 695.913017] LNet: 97549:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 695.923156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 695.948341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 695.956022] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 696.250821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 696.257695] LNetError: 97698:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 696.262451] LNet: Removed LNI 192.168.203.5@tcp [ 696.270221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 696.286702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 696.295479] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 696.449187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 696.458860] LNet: 97726:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 696.463114] LNet: 97726:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 696.470474] LNet: Removed LNI 192.168.203.5@tcp [ 696.477529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 696.500893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 696.511354] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 698.752758] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 13:52:21 (1782928341) [ 698.785918] LNet: 98309:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 698.788591] LNetError: 98309:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 698.791195] LNetError: 98309:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 698.801533] LNet: Removed LNI 192.168.203.5@tcp [ 699.045283] Key type .llcrypt unregistered [ 699.046385] Key type ._llcrypt unregistered [ 701.094616] Key type ._llcrypt registered [ 701.096449] Key type .llcrypt registered [ 701.110842] LNet: 98546:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 701.110842] mask usage: [+|-] ... [ 701.164702] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 701.170181] alg: No test for adler32 (adler32-zlib) [ 701.973454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 702.004260] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 702.005917] LNet: *** cfs_fail_loc=e003, val=100*** [ 702.260658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.5@tcp [ 702.269182] Ping reply received with mapping: [ 702.270405] found metadata reply NI[192.168.203.5@tcp] [ 702.271520] 05 cb a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 702.272766] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 702.274070] 5a 5a 5a 5a 5a 5a 5a 5a [ 702.322435] LNet: 98712:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 702.325933] LNetError: 98712:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 702.335311] LNet: Removed LNI 192.168.203.5@tcp [ 702.566446] Key type .llcrypt unregistered [ 702.567632] Key type ._llcrypt unregistered [ 706.409124] Key type ._llcrypt registered [ 706.410521] Key type .llcrypt registered [ 706.911283] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 13:52:30 (1782928350) [ 707.078334] Key type .llcrypt unregistered [ 707.079589] Key type ._llcrypt unregistered [ 709.326464] Key type ._llcrypt registered [ 709.327575] Key type .llcrypt registered [ 709.340587] LNet: 99706:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 709.340587] mask usage: [+|-] ... [ 709.403647] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 709.409739] alg: No test for adler32 (adler32-zlib) [ 710.277061] LNetError: 99757:0:(api-ni.c:2779:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 710.413329] Key type .llcrypt unregistered [ 710.415362] Key type ._llcrypt unregistered [ 712.050486] Key type ._llcrypt registered [ 712.052370] Key type .llcrypt registered [ 712.502359] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 13:52:35 (1782928355) [ 712.671411] Key type .llcrypt unregistered [ 712.672485] Key type ._llcrypt unregistered [ 714.627708] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 714.630442] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 716.802982] Key type ._llcrypt registered [ 716.803955] Key type .llcrypt registered [ 716.816375] LNet: 100623:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 716.816375] mask usage: [+|-] ... [ 716.868405] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 716.874945] alg: No test for adler32 (adler32-zlib) [ 717.675513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 717.694869] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 721.374959] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 13:52:44 (1782928364) [ 721.418695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 721.428792] LNet: 101534:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 721.432934] LNetError: 101534:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 721.446377] LNet: Removed LNI 10.1.2.3@tcp [ 721.453980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 721.688230] Key type .llcrypt unregistered [ 721.689855] Key type ._llcrypt unregistered [ 723.711926] Key type ._llcrypt registered [ 723.713010] Key type .llcrypt registered [ 723.726925] LNet: 101766:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 723.726925] mask usage: [+|-] ... [ 723.774641] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 723.782675] alg: No test for adler32 (adler32-zlib) [ 724.589280] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 724.610355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.5@tcp [ 724.662542] LNet: 101823:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 724.666670] LNetError: 101823:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 724.675321] LNet: Removed LNI 192.168.203.5@tcp [ 724.915191] Key type .llcrypt unregistered [ 724.916321] Key type ._llcrypt unregistered [ 728.555887] Key type ._llcrypt registered [ 728.557655] Key type .llcrypt registered [ 729.011696] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 13:52:52 (1782928372) [ 729.206298] Key type .llcrypt unregistered [ 729.207625] Key type ._llcrypt unregistered [ 731.462033] Key type ._llcrypt registered [ 731.463871] Key type .llcrypt registered [ 731.477912] LNet: 102805:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 731.477912] mask usage: [+|-] ... [ 731.527768] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 731.533058] alg: No test for adler32 (adler32-zlib) [ 734.336722] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 13:52:57 (1782928377) [ 734.372526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 734.392962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 734.421513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 734.430890] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 734.694728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 734.705830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 734.722480] LNet: Removed LNI 192.168.203.5@tcp [ 734.810668] LNet: 103563:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 734.959590] Key type .llcrypt unregistered [ 734.961136] Key type ._llcrypt unregistered [ 738.784630] Key type ._llcrypt registered [ 738.785676] Key type .llcrypt registered [ 739.249393] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 13:53:02 (1782928382) [ 739.316280] LNet: 104327:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 739.316280] mask usage: [+|-] ... [ 739.366342] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 739.371382] alg: No test for adler32 (adler32-zlib) [ 740.176782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 740.195088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 740.202836] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 740.446681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.5@tcp [ 740.458937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.203.5@tcp [ 740.466209] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 740.467817] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 740.469156] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 740, 0, 0 [ 740.471153] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 740.472349] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 740.473781] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 740.475119] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 740, 0, 0 [ 740.476981] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 740.478157] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 740.479776] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 740.481168] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 740, 0, 790 [ 740.483371] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 740.484586] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 740.486126] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 740.487589] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 740, 0, 790 [ 740.489444] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 742.553260] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 13:53:05 (1782928385) [ 742.688151] LNet: 105101:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 742.691214] LNetError: 105101:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 742.698368] LNet: Removed LNI 192.168.203.5@tcp [ 742.950552] Key type .llcrypt unregistered [ 742.952043] Key type ._llcrypt unregistered [ 744.984113] Key type ._llcrypt registered [ 744.985365] Key type .llcrypt registered [ 744.998845] LNet: 105323:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 744.998845] mask usage: [+|-] ... [ 745.054844] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 745.060668] alg: No test for adler32 (adler32-zlib) [ 745.860652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 745.871279] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 748.755095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.105@tcp [ 751.549447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 751.560329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 751.568306] LNet: Added LNI 192.168.203.5@tcp1 [8/256/0/180] [ 751.874439] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 751.906176] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 751.908260] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 751.911357] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 754.058892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 754.069327] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 757.411973] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 13:53:20 (1782928400) [ 757.452491] LNet: 106602:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 757.455196] LNetError: 106602:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 757.458669] LNet: Removed LNI 192.168.203.5@tcp [ 758.496151] LNet: Removed LNI 192.168.203.5@tcp1 [ 758.498069] LNetError: 106602:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 758.500791] LNetError: 106602:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 758.508575] LNet: Removed LNI 10.1.2.3@tcp99 [ 758.721359] Key type .llcrypt unregistered [ 758.723112] Key type ._llcrypt unregistered [ 760.715924] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 760.717744] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 762.873864] Key type ._llcrypt registered [ 762.874915] Key type .llcrypt registered [ 762.890410] LNet: 106870:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 762.890410] mask usage: [+|-] ... [ 762.940641] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 762.945795] alg: No test for adler32 (adler32-zlib) [ 763.745247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 763.765951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 763.774713] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 764.023766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 764.033181] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 764.751203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 764.768461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 764.784360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 764.794165] LNetError: 107349:0:(peer.c:2234:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 764.807655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 764.827743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.5@tcp [ 764.984254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 765.016607] LNet: 107369:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 765.019900] LNetError: 107369:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 765.023190] LNet: Removed LNI 10.1.2.3@tcp [ 766.055472] LNet: Removed LNI 192.168.203.5@tcp [ 766.288193] Key type .llcrypt unregistered [ 766.289223] Key type ._llcrypt unregistered [ 769.935622] Key type ._llcrypt registered [ 769.936632] Key type .llcrypt registered [ 770.362727] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 13:53:33 (1782928413) [ 770.449483] LNet: 108181:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 770.449483] mask usage: [+|-] ... [ 770.509897] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 770.515766] alg: No test for adler32 (adler32-zlib) [ 771.323363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 771.340189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 771.349529] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 771.607643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.5@tcp [ 771.639403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg305-client@tcp [ 773.690575] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 13:53:36 (1782928416) [ 773.730326] LNet: 108912:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 773.732642] LNetError: 108912:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 773.742378] LNet: Removed LNI 192.168.203.5@tcp [ 773.954297] Key type .llcrypt unregistered [ 773.955344] Key type ._llcrypt unregistered [ 775.814355] Key type ._llcrypt registered [ 775.816224] Key type .llcrypt registered [ 775.868550] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 775.873979] alg: No test for adler32 (adler32-zlib) [ 776.669885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 776.679898] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 776.690010] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 778.677797] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 13:53:41 (1782928421) [ 779.128300] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 779.629869] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 13:53:42 (1782928422) [ 780.101679] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 780.624928] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 13:53:43 (1782928423) [ 780.655526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 780.666146] LNet: 110328:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 780.669452] LNetError: 110328:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 780.678550] LNet: Removed LNI 192.168.203.5@tcp [ 780.691201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 780.709244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 780.716511] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 780.959034] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 780.975594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 783.054785] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 13:53:46 (1782928426) [ 783.091196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 783.100153] LNet: 111038:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 783.102242] LNetError: 111038:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 783.112065] LNet: Removed LNI 192.168.203.5@tcp [ 783.120066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 783.140299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 783.150572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 783.374213] Key type .llcrypt unregistered [ 783.375322] Key type ._llcrypt unregistered [ 786.949927] Key type ._llcrypt registered [ 786.951296] Key type .llcrypt registered [ 787.387872] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 13:53:50 (1782928430) [ 787.458380] LNet: 111846:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 787.458380] mask usage: [+|-] ... [ 787.508781] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 787.513972] alg: No test for adler32 (adler32-zlib) [ 788.315655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 788.336446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 788.345133] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 788.563939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 788.584612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.5@tcp [ 790.599602] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 13:53:53 (1782928433) [ 790.641682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 790.650917] LNet: 112592:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 790.654933] LNetError: 112592:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 790.665867] LNet: Removed LNI 192.168.203.5@tcp [ 790.674988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 790.697142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 790.709472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 792.716027] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 13:53:56 (1782928436) [ 792.745858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 792.754761] LNet: 113169:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 792.765711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 792.789848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 792.903832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 792.915302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 792.934079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 793.058655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 793.069229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 793.091246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 793.103744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 793.124122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 793.242162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 793.254302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 793.272971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 793.385865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 793.396141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 795.340457] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 13:53:58 (1782928438) [ 795.369442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 795.379906] LNet: 113801:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 795.382362] LNet: 113801:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 795.392489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 795.414747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 795.422676] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 795.655018] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 795.662798] LNet: Added LNI 192.168.203.5@tcp2 [8/256/0/180] [ 795.950326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 795.969832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.5@tcp [ 796.039082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 796.045519] LNetError: 114085:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 796.047511] LNet: Removed LNI 192.168.203.5@tcp [ 796.056263] LNet: Removed LNI 192.168.203.5@tcp2 [ 796.062596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 796.080614] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 796.082907] LNet: Added LNI 192.168.203.5@tcp2 [8/256/0/180] [ 796.084519] LNet: 114091:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 796.235131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.5@tcp [ 796.303738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 797.344101] LNet: Removed LNI 192.168.203.5@tcp [ 797.345257] LNetError: 114128:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 797.347085] LNetError: 114128:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 797.355275] LNet: Removed LNI 192.168.203.5@tcp2 [ 797.363170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 797.381099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 797.389822] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 797.635694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 797.642588] LNet: Added LNI 192.168.203.5@tcp2 [8/256/0/180] [ 797.923549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.5@tcp [ 797.938048] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 797.961272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.5@tcp [ 798.036089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 798.042665] LNet: 114412:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 798.044702] LNet: 114412:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 799.072137] LNet: Removed LNI 192.168.203.5@tcp [ 800.102257] LNet: Removed LNI 192.168.203.5@tcp2 [ 800.109579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 800.130105] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 800.133370] LNet: Added LNI 192.168.203.5@tcp2 [8/256/0/180] [ 800.135359] LNet: 114419:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 800.290993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.5@tcp [ 800.353630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 800.363090] LNetError: 114456:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 800.366469] LNetError: 114456:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 800.370152] LNet: Removed LNI 192.168.203.5@tcp [ 801.382275] LNet: Removed LNI 192.168.203.5@tcp2 [ 801.389690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 801.406688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 801.417394] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 801.651541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 801.660922] LNet: Added LNI 192.168.203.5@tcp2 [8/256/0/180] [ 801.933295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 801.952935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.5@tcp [ 802.021839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 803.040152] LNet: Removed LNI 192.168.203.5@tcp [ 803.047282] LNet: Removed LNI 192.168.203.5@tcp2 [ 803.054107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 803.072595] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 803.075062] LNet: Added LNI 192.168.203.5@tcp2 [8/256/0/180] [ 803.076951] LNet: 114741:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 803.237846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.5@tcp [ 803.298828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 803.305834] LNet: 114778:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 803.309401] LNet: 114778:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 804.320112] LNet: Removed LNI 192.168.203.5@tcp [ 804.327255] LNet: Removed LNI 192.168.203.5@tcp2 [ 804.334716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 804.355205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 804.364802] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 804.605604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 804.615700] LNet: Added LNI 192.168.203.5@tcp2 [8/256/0/180] [ 804.910460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.5@tcp [ 804.924198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 804.944244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.5@tcp [ 805.013312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 805.021167] LNetError: 115062:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 805.023992] LNetError: 115062:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 805.027222] LNet: Removed LNI 192.168.203.5@tcp [ 806.054275] LNet: Removed LNI 192.168.203.5@tcp2 [ 806.061217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 806.081802] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 806.083812] LNet: Added LNI 192.168.203.5@tcp2 [8/256/0/180] [ 806.085596] LNet: 115068:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 806.235314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.5@tcp [ 806.305392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 806.315096] LNet: Removed LNI 192.168.203.5@tcp [ 807.334278] LNet: Removed LNI 192.168.203.5@tcp2 [ 807.341882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 807.364746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 807.375730] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 807.606166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 807.613425] LNet: Added LNI 192.168.203.5@tcp2 [8/256/0/180] [ 807.898444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 807.911649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 807.932409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.5@tcp [ 808.057700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 809.120119] LNet: Removed LNI 192.168.203.5@tcp [ 809.127277] LNet: Removed LNI 192.168.203.5@tcp2 [ 809.135097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 809.153294] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 809.155350] LNet: Added LNI 192.168.203.5@tcp2 [8/256/0/180] [ 809.307531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.5@tcp [ 809.416613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 810.464125] LNet: Removed LNI 192.168.203.5@tcp [ 810.471297] LNet: Removed LNI 192.168.203.5@tcp2 [ 810.480230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 810.500691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 810.508971] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 810.738964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 810.746777] LNet: Added LNI 192.168.203.5@tcp2 [8/256/0/180] [ 811.031097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.5@tcp [ 811.044786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 811.055589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 811.074193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.5@tcp [ 811.185197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 812.192143] LNet: Removed LNI 192.168.203.5@tcp [ 813.222321] LNet: Removed LNI 192.168.203.5@tcp2 [ 813.230394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 813.250383] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 813.253941] LNet: Added LNI 192.168.203.5@tcp2 [8/256/0/180] [ 813.256048] LNet: 115767:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 813.258245] LNet: 115767:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 813.413217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.5@tcp [ 815.725039] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 816.310848] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 13:54:19 (1782928459) [ 816.364486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 816.375393] LNet: 116523:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 816.378110] LNet: 116523:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 5 previous similar messages [ 816.382157] LNetError: 116523:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 816.384829] LNetError: 116523:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 9 previous similar messages [ 816.387356] LNet: Removed LNI 192.168.203.5@tcp [ 817.446518] LNet: Removed LNI 192.168.203.5@tcp2 [ 817.454189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 817.554760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 817.566032] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 817.839482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 817.849413] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 818.163433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 818.172819] LNet: Removed LNI 10.1.2.3@tcp [ 818.178247] LNet: Removed LNI 192.168.203.5@tcp [ 818.184615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 818.209959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 818.221727] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 818.223458] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 818.360643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 818.368911] LNet: Removed LNI 192.168.203.5@tcp [ 818.378496] LNet: Removed LNI 10.1.2.3@tcp [ 820.567939] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 13:54:23 (1782928463) [ 820.599489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 820.618726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 820.654769] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 820.703948] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 820.706169] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 820.708166] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 822.850681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 822.859533] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 823.130381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 823.138842] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 823.890810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.5@tcp [ 823.909731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 823.935627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.203.5@tcp -m PUT -e local_timeout -r 1 [ 823.959219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 823.979652] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 823.981724] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 824.074419] LNetError: 117457:0:(lib-move.c:4714:lnet_parse()) 192.168.203.5@tcp, src 192.168.203.5@tcp: Dropping PUT (error -108 looking up sender) [ 825.120500] LNet: Removed LNI 10.1.2.3@tcp [ 826.150249] LNet: Removed LNI 192.168.203.5@tcp [ 826.290622] Key type .llcrypt unregistered [ 826.292516] Key type ._llcrypt unregistered [ 830.045471] Key type ._llcrypt registered [ 830.046979] Key type .llcrypt registered [ 830.525603] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 13:54:33 (1782928473) [ 830.599164] LNet: 118686:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 830.599164] mask usage: [+|-] ... [ 830.654375] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 830.661641] alg: No test for adler32 (adler32-zlib) [ 831.475860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 831.600402] LNet: 118768:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 831.797276] Key type .llcrypt unregistered [ 831.798380] Key type ._llcrypt unregistered [ 833.681543] Key type ._llcrypt registered [ 833.682705] Key type .llcrypt registered [ 833.694698] LNet: 118996:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 833.694698] mask usage: [+|-] ... [ 833.745216] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 833.752899] alg: No test for adler32 (adler32-zlib) [ 834.557922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 834.567635] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 837.209025] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.105@tcp [ 837.269403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 837.489186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.203.105@tcp [ 837.573835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 837.585933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 837.598172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 837.610987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 837.623155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 837.635595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 837.647481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 837.659026] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 837.664115] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.105@tcp [ 837.666454] LNet: 119216:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 837.674107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 837.687558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 837.700102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 837.715769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 837.730212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 837.742786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 837.995316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 838.088652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 838.102535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 838.116153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 838.129491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 838.144662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 838.156132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 838.169269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 838.182923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 838.191727] LNet: 119333:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 838.201061] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 838.214251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 838.227084] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 838.244349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 838.261230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 838.274323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 838.537461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.203.105@tcp [ 838.635095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 838.646329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 838.658291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 838.671627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 838.685857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 838.698687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 838.713533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 838.726449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 838.737995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 838.754113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 838.770771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 838.784379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 838.800462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 838.815698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 839.067391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 839.164407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 839.176676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 839.189035] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 839.203299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 839.215352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 839.229290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 839.241705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 839.253949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 839.262082] LNet: 119568:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 839.265717] LNet: 119568:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 839.273656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 839.286364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 839.298685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 839.314134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 839.329805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 839.343681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 839.467381] LNet: 119614:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 839.470279] LNetError: 119614:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 839.479410] LNet: Removed LNI 192.168.203.5@tcp [ 839.633448] Key type .llcrypt unregistered [ 839.634642] Key type ._llcrypt unregistered [ 845.155931] Key type ._llcrypt registered [ 845.157655] Key type .llcrypt registered [ 845.638565] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 13:54:48 (1782928488) [ 845.722149] LNet: 120418:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 845.722149] mask usage: [+|-] ... [ 845.785494] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 845.792428] alg: No test for adler32 (adler32-zlib) [ 846.604743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 846.741880] LNet: 120499:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 846.983491] Key type .llcrypt unregistered [ 846.984746] Key type ._llcrypt unregistered [ 849.132530] Key type ._llcrypt registered [ 849.134412] Key type .llcrypt registered [ 849.148811] LNet: 120727:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 849.148811] mask usage: [+|-] ... [ 849.202254] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 849.207839] alg: No test for adler32 (adler32-zlib) [ 850.020090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 850.034324] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 852.944015] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.105@tcp [ 855.767967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 855.781970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 855.792934] LNet: Added LNI 192.168.203.5@tcp1 [8/256/0/180] [ 856.139253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 856.360250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.203.105@tcp [ 856.460375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 856.474165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 856.488633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 856.502139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 856.514937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 856.528633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 856.544693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 856.558200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 856.566084] LNet: 121149:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 856.574503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 856.588882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 856.602767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 856.615894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 856.632306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 856.645292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 856.888757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 856.977367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 856.989549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 857.002630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 857.015162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 857.028609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 857.041696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 857.055749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 857.069283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 857.076879] LNet: 121266:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 857.084694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 857.097454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 857.108331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 857.119236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 857.131490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 857.145617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 857.415202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.203.105@tcp [ 857.516122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 857.528699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 857.542252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 857.554584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 857.566838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 857.582069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 857.599144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 857.611728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 857.629293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 857.646477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 857.659845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 857.671917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 857.684782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 857.697270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 857.915615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 858.004591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 858.016131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 858.028211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 858.039847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 858.054401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 858.067037] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 858.079563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 858.095383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 858.104287] LNet: 121500:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 858.106907] LNet: 121500:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 858.114287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 858.127387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 858.141868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 858.157709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 858.170862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 858.183145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 860.434467] LNet: 121581:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 860.437389] LNetError: 121581:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 860.440415] LNet: Removed LNI 192.168.203.5@tcp [ 860.450268] LNet: Removed LNI 192.168.203.5@tcp1 [ 860.606604] Key type .llcrypt unregistered [ 860.608064] Key type ._llcrypt unregistered [ 865.959568] Key type ._llcrypt registered [ 865.960734] Key type .llcrypt registered [ 866.407489] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 13:55:09 (1782928509) [ 868.683281] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 13:55:11 (1782928511) [ 871.014132] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 13:55:14 (1782928514) [ 873.409051] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 13:55:16 (1782928516) [ 875.874939] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 13:55:19 (1782928519) [ 878.083666] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 13:55:21 (1782928521) [ 880.308097] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 13:55:23 (1782928523) [ 882.471261] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 13:55:25 (1782928525) [ 884.615049] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 13:55:27 (1782928527) [ 886.648639] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 13:55:29 (1782928529) [ 888.709844] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 13:55:31 (1782928531) [ 890.856940] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 13:55:34 (1782928534) [ 893.000845] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 13:55:36 (1782928536) [ 895.043915] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 13:55:38 (1782928538) [ 897.246699] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 13:55:40 (1782928540) [ 899.423193] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 13:55:42 (1782928542) [ 901.499170] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 13:55:44 (1782928544) [ 903.597737] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 13:55:46 (1782928546) [ 903.657018] LNet: 132104:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 903.657018] mask usage: [+|-] ... [ 903.702783] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 903.708102] alg: No test for adler32 (adler32-zlib) [ 904.504567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 904.542617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 904.549914] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 904.889812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 904.898512] LNet: 132275:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 904.900537] LNetError: 132275:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 904.906400] LNet: Removed LNI 192.168.203.5@tcp [ 904.913526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 904.932267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 904.940812] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 907.163582] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 13:55:50 (1782928550) [ 909.377244] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 13:55:52 (1782928552) [ 911.489622] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 13:55:54 (1782928554) [ 913.657565] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 13:55:56 (1782928556) [ 915.873578] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 13:55:59 (1782928559) [ 918.134916] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 13:56:01 (1782928561) [ 920.377268] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 13:56:03 (1782928563) [ 922.503121] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 13:56:05 (1782928565) [ 924.574235] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 13:56:07 (1782928567) [ 926.726763] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 13:56:10 (1782928570) [ 928.863988] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 13:56:12 (1782928572) [ 930.890377] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 13:56:14 (1782928574) [ 932.835378] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 825 sec =========== 13:56:16 (1782928576) [ 932.912084] LNet: 139731:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 932.914450] LNetError: 139731:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 932.923484] LNet: Removed LNI 192.168.203.5@tcp [ 933.118307] Key type .llcrypt unregistered [ 933.119374] Key type ._llcrypt unregistered [ 936.707906] Key type ._llcrypt registered [ 936.709314] Key type .llcrypt registered [ 936.806309] LNet: 140206:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 936.806309] mask usage: [+|-] ... [ 936.910446] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 936.915684] alg: No test for adler32 (adler32-zlib) [ 937.781714] Lustre: Lustre: Build Version: 2.17.54_83_g6b808a3 [ 937.874637] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 939.464193] Key type lgssc registered [ 939.866863] Lustre: Echo OBD driver; http://www.lustre.org/ [ 961.749986] Lustre: Mounted lustre-client [ 961.792905] LNet: 141666:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 961.792905] mask usage: [+|-] ... [ 963.961194] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 966.097629] Lustre: Unmounted lustre-client [ 977.631751] Key type lgssc unregistered [ 977.746487] LNet: 142646:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 977.748746] LNetError: 142646:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 977.759360] LNet: Removed LNI 192.168.203.5@tcp [ 977.974093] Key type .llcrypt unregistered [ 977.974954] Key type ._llcrypt unregistered