[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 378375483 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001010] APIC: Switch to symmetric I/O mode setup [ 0.002362] x2apic enabled [ 0.003004] Switched APIC routing to physical x2apic. [ 0.004010] kvm-guest: setup PV IPIs [ 0.006414] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007019] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008009] pid_max: default: 32768 minimum: 301 [ 0.009112] LSM: Security Framework initializing [ 0.010035] Yama: becoming mindful. [ 0.011028] SELinux: Initializing. [ 0.012063] *** VALIDATE selinux *** [ 0.020443] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.024698] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.025131] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.026089] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027102] *** VALIDATE tmpfs *** [ 0.028388] *** VALIDATE proc *** [ 0.029204] *** VALIDATE cgroup *** [ 0.030005] *** VALIDATE cgroup2 *** [ 0.031234] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.032104] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.033003] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.034019] Spectre V2 : User space: Vulnerable [ 0.035003] Speculative Store Bypass: Vulnerable [ 0.037370] debug: unmapping init [mem 0xffffffffb8e59000-0xffffffffb8e60fff] [ 0.039917] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.040659] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.041020] ... version: 2 [ 0.041726] ... bit width: 48 [ 0.042008] ... generic registers: 4 [ 0.042983] ... value mask: 0000ffffffffffff [ 0.043011] ... max period: 00007fffffffffff [ 0.044006] ... fixed-purpose events: 3 [ 0.044985] ... event mask: 000000070000000f [ 0.045221] rcu: Hierarchical SRCU implementation. [ 0.046961] smp: Bringing up secondary CPUs ... [ 0.047354] x86: Booting SMP configuration: [ 0.048013] .... node #0, CPUs: #1 #2 #3 [ 0.050696] smp: Brought up 1 node, 4 CPUs [ 0.051763] smpboot: Max logical packages: 1 [ 0.052010] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.132015] node 0 deferred pages initialised in 79ms [ 0.134507] devtmpfs: initialized [ 0.135133] x86/mm: Memory block size: 128MB [ 0.137233] gcov: version magic: 0x41383552 [ 0.138551] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.139046] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.140162] pinctrl core: initialized pinctrl subsystem [ 0.141106] [ 0.141394] ************************************************************* [ 0.142006] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.143007] ** ** [ 0.144005] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.145005] ** ** [ 0.146005] ** This means that this kernel is built to expose internal ** [ 0.147005] ** IOMMU data structures, which may compromise security on ** [ 0.148007] ** your system. ** [ 0.149005] ** ** [ 0.150008] ** If you see this message and you are not debugging the ** [ 0.151009] ** kernel, report this immediately to your vendor! ** [ 0.152006] ** ** [ 0.153006] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.154007] ************************************************************* [ 0.155577] NET: Registered protocol family 16 [ 0.156397] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.157043] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.158034] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.159422] cpuidle: using governor menu [ 0.160476] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.162388] PCI: Using configuration type 1 for base access [ 0.164087] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.172092] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.173015] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.174130] cryptd: max_cpu_qlen set to 1000 [ 0.176311] ACPI: Added _OSI(Module Device) [ 0.177011] ACPI: Added _OSI(Processor Device) [ 0.178007] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.179008] ACPI: Added _OSI(Processor Aggregator Device) [ 0.182059] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.187515] ACPI: Interpreter enabled [ 0.188045] ACPI: PM: (supports S0 S3 S4 S5) [ 0.189011] ACPI: Using IOAPIC for interrupt routing [ 0.190072] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.193260] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.201562] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.203024] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.205012] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.207057] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.211133] acpiphp: Slot [2] registered [ 0.212030] acpiphp: Slot [5] registered [ 0.212914] acpiphp: Slot [6] registered [ 0.214063] acpiphp: Slot [3] registered [ 0.214964] acpiphp: Slot [4] registered [ 0.216047] acpiphp: Slot [7] registered [ 0.216967] acpiphp: Slot [8] registered [ 0.218056] acpiphp: Slot [9] registered [ 0.219058] acpiphp: Slot [10] registered [ 0.220083] acpiphp: Slot [11] registered [ 0.221063] acpiphp: Slot [12] registered [ 0.222081] acpiphp: Slot [13] registered [ 0.224060] acpiphp: Slot [14] registered [ 0.225069] acpiphp: Slot [15] registered [ 0.226059] acpiphp: Slot [16] registered [ 0.227071] acpiphp: Slot [17] registered [ 0.228048] acpiphp: Slot [18] registered [ 0.229033] acpiphp: Slot [19] registered [ 0.230068] acpiphp: Slot [20] registered [ 0.231042] acpiphp: Slot [21] registered [ 0.232049] acpiphp: Slot [22] registered [ 0.233026] acpiphp: Slot [23] registered [ 0.234067] acpiphp: Slot [24] registered [ 0.234982] acpiphp: Slot [25] registered [ 0.236059] acpiphp: Slot [26] registered [ 0.237046] acpiphp: Slot [27] registered [ 0.238079] acpiphp: Slot [28] registered [ 0.238910] acpiphp: Slot [29] registered [ 0.240081] acpiphp: Slot [30] registered [ 0.241012] acpiphp: Slot [31] registered [ 0.242072] PCI host bridge to bus 0000:00 [ 0.243012] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.244009] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.246012] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.248015] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.250012] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.252018] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.253162] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.255878] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.258112] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.265015] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.269076] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.271012] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.273010] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.275009] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.276535] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.279876] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.283042] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.286227] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.290945] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.300425] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.301784] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.305110] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.307866] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.316019] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.326015] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.336637] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.344015] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.348014] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.358012] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.366972] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.369576] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.372344] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.375340] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.377162] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.381097] iommu: Default domain type: Passthrough [ 0.383401] SCSI subsystem initialized [ 0.384076] ACPI: bus type USB registered [ 0.385067] usbcore: registered new interface driver usbfs [ 0.387075] usbcore: registered new interface driver hub [ 0.389068] usbcore: registered new device driver usb [ 0.391111] pps_core: LinuxPPS API ver. 1 registered [ 0.392006] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.395047] PTP clock support registered [ 0.397062] EDAC MC: Ver: 3.0.0 [ 0.398341] PCI: Using ACPI for IRQ routing [ 0.399441] NetLabel: Initializing [ 0.400009] NetLabel: domain hash size = 128 [ 0.402008] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.403069] NetLabel: unlabeled traffic allowed by default [ 0.405099] vgaarb: loaded [ 0.407242] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.408006] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.413295] clocksource: Switched to clocksource kvm-clock [ 0.502295] VFS: Disk quotas dquot_6.6.0 [ 0.503172] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.504635] *** VALIDATE ramfs *** [ 0.505471] *** VALIDATE hugetlbfs *** [ 0.506318] pnp: PnP ACPI init [ 0.507920] pnp: PnP ACPI: found 6 devices [ 0.521119] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.523791] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.526244] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.527852] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.529270] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.531226] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.532995] NET: Registered protocol family 2 [ 0.535103] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.538337] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.540511] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.544520] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.547147] TCP: Hash tables configured (established 65536 bind 65536) [ 0.549441] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.551394] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.553258] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.555288] NET: Registered protocol family 1 [ 0.556827] RPC: Registered named UNIX socket transport module. [ 0.558623] RPC: Registered udp transport module. [ 0.559724] RPC: Registered tcp transport module. [ 0.560806] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.562356] NET: Registered protocol family 44 [ 0.563318] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.565268] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.567296] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.569414] PCI: CLS 0 bytes, default 64 [ 0.570513] Unpacking initramfs... [ 1.823765] debug: unmapping init [mem 0xffff90567cc64000-0xffff90567ffcffff] [ 1.827093] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 1.828543] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 1.831187] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.276876] Initialise system trusted keyrings [ 2.278082] Key type blacklist registered [ 2.279367] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.287148] zbud: loaded [ 2.289438] *** VALIDATE nfs *** [ 2.290087] *** VALIDATE nfs4 *** [ 2.291196] pstore: using deflate compression [ 2.293484] Platform Keyring initialized [ 2.366101] NET: Registered protocol family 38 [ 2.369066] Key type asymmetric registered [ 2.370527] Asymmetric key parser 'x509' registered [ 2.372075] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.374446] io scheduler mq-deadline registered [ 2.375451] io scheduler kyber registered [ 2.376431] io scheduler bfq registered [ 2.377528] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.379412] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.381533] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.384255] ACPI: Power Button [PWRF] [ 2.388687] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.393464] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.401869] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.426982] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.454073] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.458119] Non-volatile memory driver v1.3 [ 2.459329] Linux agpgart interface v0.103 [ 2.483163] virtio_blk virtio1: [vda] 134856 512-byte logical blocks (69.0 MB/65.8 MiB) [ 2.485443] vda: detected capacity change from 0 to 69046272 [ 2.496469] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 2.498654] vdb: detected capacity change from 0 to 1073741824 [ 2.505337] libphy: Fixed MDIO Bus: probed [ 2.508561] usbcore: registered new interface driver usbserial_generic [ 2.510639] usbserial: USB Serial support registered for generic [ 2.513035] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 2.516754] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 2.518565] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 2.520735] mousedev: PS/2 mouse device common for all mice [ 2.523067] rtc_cmos 00:05: RTC can wake from S4 [ 2.524445] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 2.525573] rtc_cmos 00:05: registered as rtc0 [ 2.527740] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 2.530057] intel_pstate: CPU model not supported [ 2.532667] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 2.533983] hid: raw HID events driver (C) Jiri Kosina [ 2.537065] usbcore: registered new interface driver usbhid [ 2.538478] usbhid: USB HID core driver [ 2.538574] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 2.539448] drop_monitor: Initializing network drop monitor service [ 2.542711] Initializing XFRM netlink socket [ 2.544056] NET: Registered protocol family 10 [ 2.545660] Segment Routing with IPv6 [ 2.547135] NET: Registered protocol family 17 [ 2.548868] mpls_gso: MPLS GSO support [ 2.552794] RAS: Correctable Errors collector initialized. [ 2.554396] AVX version of gcm_enc/dec engaged. [ 2.555428] AES CTR mode by8 optimization enabled [ 2.604402] sched_clock: Marking stable (2604385321, 0)->(3339767511, -735382190) [ 2.607138] registered taskstats version 1 [ 2.608454] Loading compiled-in X.509 certificates [ 2.610298] zswap: loaded using pool lzo/zbud [ 2.629054] Key type big_key registered [ 2.637650] Key type encrypted registered [ 2.639061] ima: No TPM chip found, activating TPM-bypass! [ 2.640309] ima: Allocated hash algorithm: sha1 [ 2.641629] ima: No architecture policies found [ 2.642729] evm: Initialising EVM extended attributes: [ 2.643827] evm: security.selinux [ 2.644620] evm: security.ima [ 2.645291] evm: security.capability [ 2.646194] evm: HMAC attrs: 0x1 [ 2.647920] rtc_cmos 00:05: setting system clock to 2026-06-01 16:23:33 UTC (1780331013) [ 2.652162] debug: unmapping init [mem 0xffffffffb9e03000-0xffffffffb9ffffff] [ 2.654381] debug: unmapping init [mem 0xffffffffb8b82000-0xffffffffb8e58fff] [ 2.662064] Write protecting the kernel read-only data: 28672k [ 2.665370] debug: unmapping init [mem 0xffffffffb7203000-0xffffffffb73fffff] [ 2.667863] debug: unmapping init [mem 0xffffffffb7b14000-0xffffffffb7bfffff] [ 2.690617] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 2.696481] systemd[1]: Detected virtualization kvm. [ 2.697816] systemd[1]: Detected architecture x86-64. [ 2.699169] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 2.722337] systemd[1]: No hostname configured. [ 2.723589] systemd[1]: Set hostname to . [ 2.724701] random: systemd: uninitialized urandom read (16 bytes read) [ 2.726035] systemd[1]: Initializing machine ID from random generator. [ 2.762132] random: ln: uninitialized urandom read (6 bytes read) [ 2.817115] random: systemd: uninitialized urandom read (16 bytes read) [ 2.819041] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 2.822727] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 2.825929] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Slices. Starting Setup Virtual Console... [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Swap. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. Starting Apply Kernel Variables... [ OK ] Reached target Initrd Root Device. Starting Journal Service... [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. Starting Create Volatile Files and Directories... [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 3.252345] device-mapper: uevent: version 1.0.3 [ 3.253734] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 3.800259] virtio_net virtio0 ens2: renamed from eth0 [ 3.832490] scsi host0: ata_piix [ 3.892091] scsi host1: ata_piix [ 3.893199] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 3.894662] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 7.363496] dracut-initqueue[591]: RTNETLINK answers: File exists [ 8.149710] random: crng init done [ 8.150503] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 8.467591] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Timers. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Apply Kernel Variables. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 9.277942] printk: systemd: 26 output lines suppressed due to ratelimiting [ 9.435398] SELinux: Disabled at runtime. [ 9.482470] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 9.490709] systemd[1]: Detected virtualization kvm. [ 9.492386] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 9.815551] systemd[1]: initrd-switch-root.service: Succeeded. [ 9.818461] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 9.824293] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 9.827028] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 9.828730] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 9.833521] systemd[1]: Starting Journal Service... Starting Journal Service... [ 9.836454] systemd[1]: Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on udev Control Socket. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Mounting POSIX Message Queue File System... [ OK ] Listening on Process Core Dump Socket. Mounting Kernel Debug File System... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on RPCbind Server Activation Socket. Mounting Huge Pages File System... Starting Remount Root and Kernel File Systems... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Stopped target Initrd File Systems. Starting Apply Kernel Variables... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Created slice system-getty.slice. [ OK ] Reached target RPC Port Mapper. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... Activating swap /dev/disk/by-label/SWAP... [ OK ] Started Jour[ 9.958885] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS nal Service. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Huge Pages File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ 10.147560] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ OK ] Started udev Coldplug all Devices. [ 10.405793] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 10.454198] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 10.459463] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 10.470923] EDAC sbridge: Ver: 1.1.2 [ 11.260664] Key type dns_resolver registered [ 11.519776] NFS: Registering the id_resolver key type [ 11.520849] Key type id_resolver registered [ 11.521645] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Login Service... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg126-client login: [ 32.858524] libcfs: loading out-of-tree module taints kernel. [ 32.877285] Key type ._llcrypt registered [ 32.878209] Key type .llcrypt registered [ 33.162637] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 33.168037] alg: No test for adler32 (adler32-zlib) [ 34.127149] Lustre: Lustre: Build Version: 2.17.53_31_g0c7f08d [ 34.390980] LNet: Added LNI 192.168.201.26@tcp [8/256/0/180] [ 35.999143] Key type lgssc registered [ 36.545457] Lustre: Echo OBD driver; http://www.lustre.org/ [ 80.874863] Lustre: Mounted lustre-client [ 83.338455] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 92.762296] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing check_logdir /tmp/testlogs/ [ 94.073476] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing yml_node [ 95.416251] Lustre: DEBUG MARKER: Client: 2.17.53.31 [ 96.224919] Lustre: DEBUG MARKER: MDS: 2.17.53.31 [ 97.028747] Lustre: DEBUG MARKER: OSS: 2.17.53.31 [ 97.523428] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Mon Jun 1 12:25:07 EDT 2026 [ 101.925235] LustreError: 7943:0:(lov_obd.c:792:lov_cleanup()) lustre-clilov-ffff9056c9f39000: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 101.936932] Lustre: Unmounted lustre-client [ 112.631807] Key type lgssc unregistered [ 112.729417] LNet: 8388:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 112.731559] LNetError: 8388:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 112.738580] LNet: Removed LNI 192.168.201.26@tcp [ 112.955091] Key type .llcrypt unregistered [ 112.956557] Key type ._llcrypt unregistered [ 120.457685] Key type ._llcrypt registered [ 120.459212] Key type .llcrypt registered [ 120.524691] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 120.529602] alg: No test for adler32 (adler32-zlib) [ 121.326311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 121.349277] LNet: Added LNI 192.168.201.26@tcp [8/256/0/180] [ 121.356059] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 122.608494] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 122.635664] LNet: 9258:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 122.638381] LNetError: 9258:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 122.644355] LNet: Removed LNI 192.168.201.26@tcp [ 122.859308] Key type .llcrypt unregistered [ 122.860487] Key type ._llcrypt unregistered [ 124.704473] Key type ._llcrypt registered [ 124.705707] Key type .llcrypt registered [ 125.128782] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 12:25:35 (1780331135) [ 125.281771] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 125.288403] alg: No test for adler32 (adler32-zlib) [ 126.083364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 126.108810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 126.115186] LNet: 9675:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 126.122544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 126.139797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 126.145745] LNet: 9682:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 128.299852] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 12:25:38 (1780331138) [ 128.336688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 128.348437] LNet: 10253:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 128.359371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 128.393663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 130.371896] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 12:25:40 (1780331140) [ 130.401834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 130.411076] LNet: 10839:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 130.422238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 130.448814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 132.527941] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 12:25:42 (1780331142) [ 132.560528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 132.570207] LNet: 11420:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 132.580868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 132.617284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 134.900270] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 12:25:45 (1780331145) [ 134.939531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 134.960280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 134.990445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 135.118856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 137.270079] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 12:25:47 (1780331147) [ 137.300966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 137.308874] LNet: 12594:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 137.311028] LNet: 12594:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 137.322300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 137.354140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 137.486210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 137.625019] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 139.933697] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 12:25:50 (1780331150) [ 140.007136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 140.023748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 140.054194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 142.407261] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 12:25:52 (1780331152) [ 142.437083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 142.454650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 142.484895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 142.500262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 142.641279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 142.657578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 142.817732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 142.832444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 142.984587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 142.999942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 143.156514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 143.172054] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 143.317971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 143.333953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 143.478596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 143.492701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 143.671408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 143.688735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 143.839688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 143.863506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 144.056338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 144.071765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 144.266687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 144.285062] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 144.440170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 144.470200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 144.654714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 144.674950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 148.471343] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 12:25:58 (1780331158) [ 148.681672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 148.697879] LNet: 14459:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 148.701065] LNet: 14459:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 148.713761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 148.787267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 148.817603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 152.895422] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 12:26:02 (1780331162) [ 153.003722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 153.076058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 153.127892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 153.202425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 159.295842] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 12:26:09 (1780331169) [ 159.500229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 159.533930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 159.576798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 159.601919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 164.090463] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 12:26:13 (1780331173) [ 164.276420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 164.304729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 164.370357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 164.398880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 168.713772] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 12:26:18 (1780331178) [ 168.860943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 168.910565] LNet: 16802:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 168.916875] LNet: 16802:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 168.939529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 169.014573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 169.050742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 173.629994] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 12:26:23 (1780331183) [ 173.691246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 173.713146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 173.778615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 173.801797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 178.796313] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 12:26:28 (1780331188) [ 178.949912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 178.977281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 179.038909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 179.294885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 184.536174] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 12:26:34 (1780331194) [ 184.885289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 184.940961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 185.219678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 185.410193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 199.686396] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 12:26:46 (1780331206) [ 200.039779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 200.093910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 200.494573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 201.124685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 210.881341] hrtimer: interrupt took 2534359 ns [ 219.042789] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 12:27:06 (1780331226) [ 219.281647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 219.335937] LNet: 19800:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 219.359426] LNet: 19800:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 219.447722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 220.955756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 221.635394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 235.629859] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 12:27:24 (1780331244) [ 235.797352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 235.890652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 236.155552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 236.371712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 249.300843] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 12:27:38 (1780331258) [ 250.212386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 250.383632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 250.554159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 250.663645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 258.997644] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 12:27:48 (1780331268) [ 259.307527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 259.361849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 259.435950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 266.309172] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 12:27:55 (1780331275) [ 266.558235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 266.630041] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 266.746873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 272.815949] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 12:28:02 (1780331282) [ 273.059459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 273.093750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 273.209835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 273.681380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 279.866899] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 12:28:09 (1780331289) [ 279.954439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 280.009886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 280.151932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 280.188779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 286.308511] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 12:28:15 (1780331295) [ 286.391847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 286.415139] LNet: 24003:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 286.422569] LNet: 24003:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 6 previous similar messages [ 286.462404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 286.531258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 286.572358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 292.226583] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 12:28:21 (1780331301) [ 292.359968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 292.422809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 292.538596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 292.566556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 297.466316] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 12:28:27 (1780331307) [ 297.591556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 297.608772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 297.656588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 297.684721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 303.163562] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 12:28:32 (1780331312) [ 303.219139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 303.250674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 303.300658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 303.365622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 303.386123] LNetError: 25779:0:(peer.c:2175:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 303.410963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 303.453374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 308.307599] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 12:28:38 (1780331318) [ 308.467746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 308.499540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 308.564352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.TGJrbfyDPF [ 308.608750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.TGJrbfyDPF [ 308.638822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.TGJrbfyDPF [ 314.028522] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 12:28:43 (1780331323) [ 314.096379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 314.135375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 314.175912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 314.193728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 314.228749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 319.525267] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 12:28:49 (1780331329) [ 319.614584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 319.673993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 319.747751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 321.898524] LNet: Message forwarding will be enabled in 2s [ 324.959129] LNet: Message forwarding enabled [ 325.280943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 325.316611] LNet: Message forwarding will stop in 2s to 6s [ 329.861803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 330.604120] LNet: Message forwarding disabled [ 332.081113] LNet: Message forwarding will be enabled in 2s [ 335.711132] LNet: Message forwarding enabled [ 337.719893] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 12:29:07 (1780331347) [ 337.867917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 338.946106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 339.001392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 344.684929] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 12:29:14 (1780331354) [ 344.944588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 345.002686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 345.070866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 345.110792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 345.152943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 345.179856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 345.211606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 345.244528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 345.280336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 345.310870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 345.348883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 345.371591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 345.399741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 345.426976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 345.463823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 345.511883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 345.571232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 345.645714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 345.695714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 345.728434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 345.757894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 345.784821] LNetError: 28850:0:(router.c:624:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 351.382275] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 12:29:20 (1780331360) [ 351.459331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 351.498514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 351.568559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 357.709794] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 12:29:27 (1780331367) [ 359.239721] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg126-server [ 360.791560] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 12:29:30 (1780331370) [ 361.029423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 361.092531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 361.173501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 361.190908] LNet: Added LNI 192.168.201.26@tcp [8/256/0/180] [ 362.615776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.201.[27-31]@tcp [ 362.636885] LNet: 30569:0:(router.c:728:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 362.686700] LNetError: 30359:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.201.27@tcp is being used as a gateway but routing feature is not turned on [ 362.709144] LNetError: 30359:0:(lib-lnet.h:1266:lnet_set_route_aliveness()) route to tcp1 through 192.168.201.27@tcp has gone from up to down [ 363.772388] LNetError: 30359:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.201.28@tcp is being used as a gateway but routing feature is not turned on [ 363.781824] LNetError: 30359:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 364.777878] LNetError: 30359:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.201.29@tcp is being used as a gateway but routing feature is not turned on [ 364.802209] LNetError: 30359:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 4 previous similar messages [ 366.857390] LNetError: 30359:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.201.29@tcp is being used as a gateway but routing feature is not turned on [ 366.868431] LNetError: 30359:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 368.982850] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 12:29:38 (1780331378) [ 370.445296] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg126-server [ 370.921611] LNetError: 30359:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.201.27@tcp is being used as a gateway but routing feature is not turned on [ 370.931822] LNetError: 30359:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 19 previous similar messages [ 371.989720] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 12:29:41 (1780331381) [ 372.163568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 372.213430] LNetError: 31504:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 372.244286] LNet: Removed LNI 192.168.201.26@tcp [ 372.294902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 372.373744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 372.411739] LNet: Added LNI 192.168.201.26@tcp [8/256/0/180] [ 372.912805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.201.[27-32/2]@tcp [ 372.930480] LNet: 31621:0:(router.c:728:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 372.941898] LNet: 31621:0:(router.c:728:lnet_add_route()) Skipped 4 previous similar messages [ 372.960390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.201.[27-32/2]@tcp [ 372.967991] LNetError: 31508:0:(lib-lnet.h:1266:lnet_set_route_aliveness()) route to tcp103 through 192.168.201.27@tcp has gone from up to down [ 372.993760] LNetError: 31508:0:(lib-lnet.h:1266:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 378.572538] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 12:29:48 (1780331388) [ 378.667633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 378.705765] LNetError: 32207:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 378.730388] LNet: Removed LNI 192.168.201.26@tcp [ 378.755734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 378.854557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 378.905876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 378.946937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 378.993988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 379.073301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 379.111116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 379.151270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 379.203488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 379.220508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 379.248370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 379.289909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 379.311708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 379.362945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 379.432910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 379.469416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 379.526690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 379.598372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 379.625059] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 379.665412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 379.729093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 379.760600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 379.825682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 379.873563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 379.913651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 379.959360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 380.018557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 380.043511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 380.090115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 380.159571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 380.189915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 380.235148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 380.298568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 380.330520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 380.392577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 385.072714] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 12:29:54 (1780331394) [ 386.573163] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg126-server [ 388.154302] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 12:29:57 (1780331397) [ 389.646926] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg126-server [ 390.937599] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 12:30:00 (1780331400) [ 391.589047] Key type .llcrypt unregistered [ 391.590620] Key type ._llcrypt unregistered [ 396.600513] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 396.689541] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 396.691892] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 396.695965] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 399.442528] Key type ._llcrypt registered [ 399.445016] Key type .llcrypt registered [ 399.662879] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 399.697641] alg: No test for adler32 (adler32-zlib) [ 400.604215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 400.645202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 400.673615] LNet: Added LNI 192.168.201.26@tcp [8/256/0/180] [ 401.271885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 401.293218] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 402.785258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 402.809686] LNet: Removed LNI 10.1.2.3@tcp [ 402.904607] LNet: 34328:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 402.907506] LNetError: 34328:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 402.915621] LNet: Removed LNI 192.168.201.26@tcp [ 403.530791] Key type .llcrypt unregistered [ 403.533071] Key type ._llcrypt unregistered [ 413.205179] Key type ._llcrypt registered [ 413.208438] Key type .llcrypt registered [ 414.221992] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 12:30:23 (1780331423) [ 414.695524] Key type .llcrypt unregistered [ 414.697793] Key type ._llcrypt unregistered [ 420.227819] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 420.360528] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 420.369240] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 420.379150] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 422.828854] Key type ._llcrypt registered [ 422.830165] Key type .llcrypt registered [ 422.987478] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 423.014701] alg: No test for adler32 (adler32-zlib) [ 423.937108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 424.010436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 424.038820] LNet: Added LNI 192.168.201.26@tcp [8/256/0/180] [ 424.592258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 424.608930] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 427.017678] LNet: 35821:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 427.022585] LNetError: 35821:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 427.027367] LNet: Removed LNI 192.168.201.26@tcp [ 427.038859] LNet: Removed LNI 10.1.2.3@tcp [ 427.484890] Key type .llcrypt unregistered [ 427.486777] Key type ._llcrypt unregistered [ 437.943762] Key type ._llcrypt registered [ 437.945517] Key type .llcrypt registered [ 439.050247] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 12:30:48 (1780331448) [ 439.439447] Key type .llcrypt unregistered [ 439.443411] Key type ._llcrypt unregistered [ 444.541318] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 444.672663] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 444.682153] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 444.693107] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 447.120470] Key type ._llcrypt registered [ 447.125273] Key type .llcrypt registered [ 447.293546] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 447.306886] alg: No test for adler32 (adler32-zlib) [ 448.208305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 448.274519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 448.308561] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 449.949847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 449.967395] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 451.544791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 451.566464] LNet: Removed LNI 10.1.2.3@tcp [ 451.578908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 451.592335] LNet: Removed LNI 10.1.2.31@tcp [ 451.604460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 451.625840] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 452.248285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 452.275205] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 453.186602] LNet: 37736:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 453.194933] LNetError: 37736:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 453.201417] LNet: Removed LNI 10.1.2.3@tcp [ 453.216153] LNet: Removed LNI 10.1.2.31@tcp [ 453.761595] Key type .llcrypt unregistered [ 453.763894] Key type ._llcrypt unregistered [ 464.502983] Key type ._llcrypt registered [ 464.504047] Key type .llcrypt registered [ 465.863970] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 12:31:15 (1780331475) [ 466.162300] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 466.176466] alg: No test for adler32 (adler32-zlib) [ 467.095324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 467.134911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 467.159974] LNet: Added LNI 192.168.201.26@tcp [8/256/0/180] [ 467.850525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 467.868224] LNet: 38688:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 467.873280] LNetError: 38688:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 467.887658] LNet: Removed LNI 192.168.201.26@tcp [ 467.906777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 467.961623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.201.26@tcp [ 468.003223] LNet: Added LNI 192.168.201.26@tcp [8/256/0/180] [ 468.677713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.201.26@tcp [ 468.713810] LNet: Removed LNI 192.168.201.26@tcp [ 468.778791] LNet: 38824:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 469.230538] Key type .llcrypt unregistered [ 469.232141] Key type ._llcrypt unregistered [ 478.686862] Key type ._llcrypt registered [ 478.689282] Key type .llcrypt registered [ 479.698706] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 12:31:29 (1780331489) [ 481.281867] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg126-server [ 482.851526] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 12:31:32 (1780331492) [ 483.228386] Key type .llcrypt unregistered [ 483.230730] Key type ._llcrypt unregistered [ 489.305375] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 489.466196] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 489.471809] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 489.477186] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 492.163740] Key type ._llcrypt registered [ 492.165508] Key type .llcrypt registered [ 492.414518] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 492.450768] alg: No test for adler32 (adler32-zlib) [ 493.374556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 493.455784] LNet: Added LNI 192.168.201.26@tcp1 [8/256/0/180] [ 494.264240] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 496.341667] LNet: 40661:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 496.346574] LNetError: 40661:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 496.354751] LNet: Removed LNI 192.168.201.26@tcp1 [ 496.363266] LNet: Removed LNI 10.1.2.3@tcp2 [ 496.975400] Key type .llcrypt unregistered [ 496.979860] Key type ._llcrypt unregistered [ 507.177985] Key type ._llcrypt registered [ 507.182242] Key type .llcrypt registered [ 508.296504] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 12:31:57 (1780331517) [ 508.796988] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 508.820179] alg: No test for adler32 (adler32-zlib) [ 509.781175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 509.814929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 509.833341] LNet: Added LNI 192.168.201.26@tcp [8/256/0/180] [ 510.679833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 510.712918] LNet: 41629:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 510.716939] LNetError: 41629:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 510.732182] LNet: Removed LNI 192.168.201.26@tcp [ 510.751260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 510.799173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 510.823747] LNet: Added LNI 192.168.201.26@tcp [8/256/0/180] [ 516.766722] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 12:32:06 (1780331526) [ 516.966737] LNet: 42218:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 516.976355] LNetError: 42218:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 517.008768] LNet: Removed LNI 192.168.201.26@tcp [ 517.660381] Key type .llcrypt unregistered [ 517.664899] Key type ._llcrypt unregistered [ 523.486429] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 523.639699] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 523.642652] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 523.645357] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 525.132955] Key type ._llcrypt registered [ 525.134768] Key type .llcrypt registered [ 525.316707] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 525.326817] alg: No test for adler32 (adler32-zlib) [ 526.230808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 526.297764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 526.313911] LNet: Added LNI 192.168.201.26@tcp [8/256/0/180] [ 526.980463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 527.013730] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 529.158501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 529.186723] LNet: 42955:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 529.191955] LNetError: 42955:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 529.196518] LNet: Removed LNI 192.168.201.26@tcp [ 529.211748] LNet: Removed LNI 10.1.2.3@tcp [ 529.233521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 529.297804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 529.345223] LNet: Added LNI 192.168.201.26@tcp [8/256/0/180] [ 529.352627] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 529.796529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 529.818989] LNet: 42982:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 529.827506] LNetError: 42982:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 529.832929] LNetError: 42982:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 529.837981] LNet: Removed LNI 192.168.201.26@tcp [ 529.847550] LNet: Removed LNI 10.1.2.3@tcp [ 529.869733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 529.964464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 530.004457] LNet: Added LNI 192.168.201.26@tcp [8/256/0/180] [ 530.014797] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 530.623736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 530.639676] LNet: Added LNI 192.168.201.26@tcp2 [8/256/0/180] [ 531.550795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 531.568696] LNet: 43144:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 531.574385] LNetError: 43144:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 531.576917] LNetError: 43144:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 531.583666] LNet: Removed LNI 192.168.201.26@tcp [ 531.585945] LNet: Removed LNI 10.1.2.3@tcp [ 531.596783] LNet: Removed LNI 192.168.201.26@tcp2 [ 531.621471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 531.668363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 531.693532] LNet: Added LNI 192.168.201.26@tcp [8/256/0/180] [ 531.698965] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 531.704440] LNet: Added LNI 192.168.201.26@tcp2 [8/256/0/180] [ 537.755915] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 12:32:27 (1780331547) [ 537.829754] LNet: 43731:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 537.834359] LNetError: 43731:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 537.840032] LNetError: 43731:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 537.844919] LNet: Removed LNI 192.168.201.26@tcp [ 537.846894] LNet: Removed LNI 10.1.2.3@tcp [ 537.856826] LNet: Removed LNI 192.168.201.26@tcp2 [ 538.389670] Key type .llcrypt unregistered [ 538.391855] Key type ._llcrypt unregistered [ 544.386722] Key type ._llcrypt registered [ 544.389445] Key type .llcrypt registered [ 544.566315] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 544.577926] alg: No test for adler32 (adler32-zlib) [ 545.505221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 545.554387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 545.580206] LNet: Added LNI 192.168.201.26@tcp [8/256/0/180] [ 546.259251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 546.270527] LNet: 44134:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 546.274504] LNetError: 44134:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 546.292621] LNet: Removed LNI 192.168.201.26@tcp [ 546.322516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 546.359526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 546.379203] LNet: Added LNI 192.168.201.26@tcp [8/256/0/180] [ 546.949537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 546.969535] LNet: 44164:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 546.974716] LNetError: 44164:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 546.990412] LNet: Removed LNI 192.168.201.26@tcp [ 547.007767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 547.075903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 547.102882] LNet: Added LNI 192.168.201.26@tcp [8/256/0/180] [ 547.583565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 547.655228] LNet: Removed LNI 192.168.201.26@tcp [ 547.702650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 547.811912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 547.851500] LNet: Added LNI 192.168.201.26@tcp [8/256/0/180] [ 548.492873] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 548.591864] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 548.595735] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 548.601700] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 551.049718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 551.096568] LNet: 44267:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 551.105913] LNet: 44267:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 551.114929] LNetError: 44267:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 551.130596] LNetError: 44267:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 551.156460] LNet: Removed LNI 192.168.201.26@tcp [ 551.228676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 551.292324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 551.319190] LNet: Added LNI 192.168.201.26@tcp [8/256/0/180] [ 558.314962] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 12:32:47 (1780331567) [ 558.444977] LNet: 44857:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 558.460525] LNetError: 44857:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 558.488229] LNet: Removed LNI 192.168.201.26@tcp [ 559.193839] Key type .llcrypt unregistered [ 559.201674] Key type ._llcrypt unregistered [ 565.275579] Key type ._llcrypt registered [ 565.278200] Key type .llcrypt registered [ 565.516975] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 565.540758] alg: No test for adler32 (adler32-zlib) [ 566.455638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 566.527841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 566.554075] LNet: Added LNI 192.168.201.26@tcp [8/256/0/180] [ 567.367455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 567.384415] LNet: 45262:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 567.389351] LNetError: 45262:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 567.409728] LNet: Removed LNI 192.168.201.26@tcp [ 567.432875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 567.482933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 567.505477] LNet: Added LNI 192.168.201.26@tcp [8/256/0/180] [ 573.116849] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 12:33:02 (1780331582) [ 573.218904] LNet: 45849:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 573.224335] LNetError: 45849:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 573.243924] LNet: Removed LNI 192.168.201.26@tcp [ 573.803541] Key type .llcrypt unregistered [ 573.805649] Key type ._llcrypt unregistered [ 579.266577] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 579.417903] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 579.421807] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 579.425074] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 581.973918] Key type ._llcrypt registered [ 581.976867] Key type .llcrypt registered [ 582.193944] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 582.209222] alg: No test for adler32 (adler32-zlib) [ 583.137441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 583.176976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 583.204964] LNet: Added LNI 192.168.201.26@tcp [8/256/0/180] [ 583.851785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 583.870336] LNet: 46304:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 583.874941] LNetError: 46304:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 583.889703] LNet: Removed LNI 192.168.201.26@tcp [ 583.915994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 583.963101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 583.982424] LNet: Added LNI 192.168.201.26@tcp [8/256/0/180] [ 590.354632] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 12:33:20 (1780331600) [ 590.600969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 590.627458] LNet: 46902:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 590.633569] LNetError: 46902:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 590.647577] LNet: Removed LNI 192.168.201.26@tcp [ 590.666029] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 590.770721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 597.216897] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 12:33:26 (1780331606) [ 597.511799] LNet: 47484:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 598.014597] Key type .llcrypt unregistered [ 598.017826] Key type ._llcrypt unregistered [ 604.480798] Key type ._llcrypt registered [ 604.485151] Key type .llcrypt registered [ 604.777918] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 604.827777] alg: No test for adler32 (adler32-zlib) [ 605.808663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 605.892704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 605.923590] LNet: Added LNI 192.168.201.26@tcp [8/256/0/180] [ 606.842789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 606.869531] LNet: 47881:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 606.882474] LNetError: 47881:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 606.909429] LNet: Removed LNI 192.168.201.26@tcp [ 606.941579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 606.994752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 607.013652] LNet: Added LNI 192.168.201.26@tcp [8/256/0/180] [ 612.804954] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 12:33:42 (1780331622) [ 612.895277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 612.933301] LNet: 48470:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 612.937524] LNetError: 48470:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 612.958833] LNet: Removed LNI 192.168.201.26@tcp [ 612.990452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 613.031172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 618.645843] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 12:33:48 (1780331628) [ 619.968733] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 621.648193] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 12:33:51 (1780331631) [ 622.772943] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 624.419879] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 12:33:54 (1780331634) [ 626.024917] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 627.677987] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 12:33:57 (1780331637) [ 629.083397] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 630.573736] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 12:34:00 (1780331640) [ 632.062824] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 633.535070] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing ========================================================== 12:34:03 (1780331643) [ 633.641285] LNet: 50517:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 634.215669] Key type .llcrypt unregistered [ 634.218136] Key type ._llcrypt unregistered [ 639.874958] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 640.023946] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 640.042136] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 640.048597] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 642.575866] Key type ._llcrypt registered [ 642.579673] Key type .llcrypt registered [ 642.842639] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 642.861662] alg: No test for adler32 (adler32-zlib) [ 643.729819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 643.790862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 643.806960] LNet: Added LNI 192.168.201.26@tcp [8/512/0/180] [ 643.813628] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 643.818575] LNet: 50845:0:(api-ni.c:2864:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 643.902598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 643.920584] LNet: 50865:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 643.926388] LNetError: 50865:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 643.929941] LNet: Removed LNI 192.168.201.26@tcp [ 643.932587] LNetError: 50865:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 643.941587] LNet: Removed LNI 10.1.2.3@tcp [ 643.955675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 644.026740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 644.065763] LNet: Added LNI 192.168.201.26@tcp [8/256/0/180] [ 644.150914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 644.172673] LNet: 50889:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 644.179851] LNetError: 50889:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 644.194405] LNet: Removed LNI 192.168.201.26@tcp [ 644.215671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 644.276706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 644.292534] LNet: Added LNI 192.168.201.26@tcp [8/256/0/180] [ 644.299904] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 644.303350] LNet: 50897:0:(api-ni.c:2864:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 644.448598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 644.472803] LNet: 50915:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 644.478755] LNetError: 50915:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 644.485754] LNet: Removed LNI 192.168.201.26@tcp [ 644.487911] LNetError: 50915:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 644.499364] LNet: Removed LNI 10.1.2.3@tcp [ 644.515850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 644.602465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 644.622331] LNet: Added LNI 192.168.201.26@tcp [8/256/0/180] [ 644.626394] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 644.631978] LNet: 50923:0:(api-ni.c:2864:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 644.837981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 644.855351] LNet: 50949:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 644.862533] LNetError: 50949:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 644.869886] LNet: Removed LNI 192.168.201.26@tcp [ 644.873472] LNetError: 50949:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 644.889777] LNet: Removed LNI 10.1.2.3@tcp [ 644.904503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 644.971215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 644.993049] LNet: Added LNI 192.168.201.26@tcp [8/256/0/182] [ 644.997050] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 645.123623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 645.144764] LNet: 50977:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 645.150978] LNetError: 50977:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 645.157241] LNet: Removed LNI 192.168.201.26@tcp [ 645.159842] LNetError: 50977:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 645.169234] LNet: Removed LNI 10.1.2.3@tcp [ 645.196911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 645.304174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 645.357532] LNet: Added LNI 192.168.201.26@tcp [8/256/0/182] [ 645.363791] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 645.370777] LNet: 50985:0:(api-ni.c:2864:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 645.446614] LNet: Removed LNI 192.168.201.26@tcp [ 645.462725] LNet: Removed LNI 10.1.2.3@tcp [ 646.100152] Key type .llcrypt unregistered [ 646.104558] Key type ._llcrypt unregistered [ 657.189442] Key type ._llcrypt registered [ 657.192389] Key type .llcrypt registered [ 658.820860] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 12:34:27 (1780331667) [ 659.193330] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 659.211239] alg: No test for adler32 (adler32-zlib) [ 660.207464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 660.262894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 660.315774] LNet: Added LNI 192.168.201.26@tcp [8/256/0/180] [ 661.352774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.201.26 [ 661.388728] LNet: 51962:0:(config.c:1628:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 661.404780] LNet: Removed LNI 192.168.201.26@tcp [ 667.631317] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 12:34:37 (1780331677) [ 667.768789] LNet: 52526:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 667.776512] LNetError: 52526:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 667.794299] LNet: Removed LNI 192.168.201.26@tcp [ 668.472359] Key type .llcrypt unregistered [ 668.473517] Key type ._llcrypt unregistered [ 674.007581] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 674.213722] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 676.891912] Key type ._llcrypt registered [ 676.893536] Key type .llcrypt registered [ 677.018651] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 677.032177] alg: No test for adler32 (adler32-zlib) [ 678.478312] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 684.874307] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 12:34:54 (1780331694) [ 685.056491] LNet: 53445:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 685.059891] LNetError: 53445:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 685.067743] LNet: Removed LNI 10.1.2.3@tcp [ 685.664759] Key type .llcrypt unregistered [ 685.673734] Key type ._llcrypt unregistered [ 692.171727] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 692.457687] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 692.462808] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 692.470480] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 695.192811] Key type ._llcrypt registered [ 695.195236] Key type .llcrypt registered [ 695.376165] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 695.388930] alg: No test for adler32 (adler32-zlib) [ 696.372398] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 702.157083] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 12:35:11 (1780331711) [ 703.555208] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 704.931668] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 12:35:14 (1780331714) [ 705.164462] LNet: 54628:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 705.177740] LNetError: 54628:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 705.222241] LNet: Removed LNI 10.1.2.3@tcp [ 705.822688] Key type .llcrypt unregistered [ 705.827178] Key type ._llcrypt unregistered [ 711.079806] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 711.258237] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 711.262524] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 711.273347] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 713.954766] Key type ._llcrypt registered [ 713.956839] Key type .llcrypt registered [ 714.082503] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 714.092696] alg: No test for adler32 (adler32-zlib) [ 715.008724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 715.065175] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 721.084928] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 12:35:31 (1780331731) [ 721.473659] LNet: 55860:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 721.476715] LNetError: 55860:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 721.487740] LNet: Removed LNI 10.1.2.3@tcp [ 721.907625] Key type .llcrypt unregistered [ 721.909600] Key type ._llcrypt unregistered [ 727.055969] Key type ._llcrypt registered [ 727.058473] Key type .llcrypt registered [ 727.194523] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 727.212747] alg: No test for adler32 (adler32-zlib) [ 728.101543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 728.123387] LNet: Added LNI 192.168.201.26@tcp [8/256/0/180] [ 733.877578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.126@tcp [ 734.444324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.26@tcp -d 192.168.201.126@tcp -m GET -e local_interrupt -r 1 [ 734.491501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.26@tcp -d 192.168.201.26@tcp -m GET -e local_interrupt -r 1 [ 734.540394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.126@tcp [ 734.569497] LNet: There was an unexpected network error while writing to 192.168.201.126: rc = -22 [ 734.769862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 734.797860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 734.824501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 734.851248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 734.882503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 734.910977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 734.931787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 734.958397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 734.970923] LNet: 56297:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 734.991122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 735.013582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 735.058632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 735.096832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 735.152858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 735.174826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 735.211605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 735.811976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.26@tcp -d 192.168.201.126@tcp -m GET -e local_dropped -r 1 [ 735.856778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.26@tcp -d 192.168.201.26@tcp -m GET -e local_dropped -r 1 [ 735.894375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.126@tcp [ 735.911666] LNet: There was an unexpected network error while writing to 192.168.201.126: rc = -22 [ 736.120986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 736.155360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 736.184378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 736.215638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 736.247729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 736.271968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 736.293676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 736.316960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 736.334591] LNet: 56406:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 736.352251] LNet: 1 local NIs in recovery (showing 1): 192.168.201.26@tcp [ 736.355368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 736.382179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 736.405314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 736.428364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 736.449446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 736.478312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 736.511308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 736.874281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.26@tcp -d 192.168.201.126@tcp -m GET -e local_aborted -r 1 [ 736.908689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.26@tcp -d 192.168.201.26@tcp -m GET -e local_aborted -r 1 [ 736.964089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.126@tcp [ 736.988301] LNet: There was an unexpected network error while writing to 192.168.201.126: rc = -22 [ 736.997667] LNet: Skipped 1 previous similar message [ 737.215649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 737.239650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 737.270466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 737.300235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 737.330962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 737.360969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 737.383774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 737.405480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 737.419158] LNet: 56516:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 737.429534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 737.452500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 737.479961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 737.506034] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 737.531151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 737.563873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 737.593526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 737.969424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.26@tcp -d 192.168.201.126@tcp -m GET -e local_no_route -r 1 [ 737.992270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.26@tcp -d 192.168.201.26@tcp -m GET -e local_no_route -r 1 [ 738.018159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.126@tcp [ 738.175422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 738.199339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 738.222471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 738.247877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 738.273603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 738.305427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 738.330915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 738.367510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 738.396428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 738.438349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 738.470195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 738.508154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 738.544620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 738.573434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 738.602836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 739.010667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.26@tcp -d 192.168.201.126@tcp -m GET -e local_timeout -r 1 [ 739.041252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.26@tcp -d 192.168.201.26@tcp -m GET -e local_timeout -r 1 [ 739.066610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.126@tcp [ 739.087863] LNet: There was an unexpected network error while writing to 192.168.201.126: rc = -22 [ 739.092219] LNet: Skipped 2 previous similar messages [ 739.268345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 739.300042] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 739.336212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 739.367841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 739.392473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 739.418331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 739.448489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 739.473333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 739.487588] LNet: 56734:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 739.499562] LNet: 56734:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 739.516435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 739.542361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 739.572365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 739.602105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 739.632739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 739.670995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 739.716386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 740.099381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.26@tcp -d 192.168.201.126@tcp -m GET -e local_error -r 1 [ 740.125811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.26@tcp -d 192.168.201.26@tcp -m GET -e local_error -r 1 [ 740.159416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.126@tcp [ 740.350227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 740.372499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 740.396370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 740.417412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 740.451240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 740.472989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 740.495684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 740.523676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 740.552947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 740.581279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 740.611393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 740.641438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 740.668327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 740.697564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 740.742406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 740.964542] LNet: 56884:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 740.972436] LNetError: 56884:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 741.991655] LNet: Removed LNI 192.168.201.26@tcp [ 742.355256] Key type .llcrypt unregistered [ 742.356518] Key type ._llcrypt unregistered [ 755.808324] Key type ._llcrypt registered [ 755.811166] Key type .llcrypt registered [ 757.555474] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 12:36:06 (1780331766) [ 758.667414] Key type .llcrypt unregistered [ 758.670497] Key type ._llcrypt unregistered [ 766.867491] Key type ._llcrypt registered [ 766.869658] Key type .llcrypt registered [ 767.152985] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 767.162855] alg: No test for adler32 (adler32-zlib) [ 768.012771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 768.059079] LNet: Added LNI 192.168.201.26@tcp [8/256/0/180] [ 777.185602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.126@tcp [ 788.187281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 788.256622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 788.303794] LNet: Added LNI 192.168.201.26@tcp1 [8/256/0/180] [ 790.750191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.26@tcp -d 192.168.201.126@tcp -m GET -e local_interrupt -r 1 [ 790.814901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.26@tcp -d 192.168.201.126@tcp1 -m GET -e local_interrupt -r 1 [ 790.857370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.26@tcp -d 192.168.201.26@tcp -m GET -e local_interrupt -r 1 [ 790.886542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.26@tcp -d 192.168.201.26@tcp1 -m GET -e local_interrupt -r 1 [ 790.949093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.26@tcp1 -d 192.168.201.126@tcp -m GET -e local_interrupt -r 1 [ 791.034991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.26@tcp1 -d 192.168.201.126@tcp1 -m GET -e local_interrupt -r 1 [ 791.107334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.26@tcp1 -d 192.168.201.26@tcp -m GET -e local_interrupt -r 1 [ 791.197305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.26@tcp1 -d 192.168.201.26@tcp1 -m GET -e local_interrupt -r 1 [ 791.304298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.126@tcp [ 791.351489] LNet: There was an unexpected network error while writing to 192.168.201.126: rc = -22 [ 791.354762] LNet: 1 local NIs in recovery (showing 1): 192.168.201.26@tcp [ 791.723505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 791.785797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 791.854334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 791.907941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 791.948723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 791.992257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 792.034157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 792.071553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 792.104299] LNet: 58353:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 792.119611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 792.144984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 792.177798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 792.234152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 792.287136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 792.324593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 792.364633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 792.821946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.26@tcp -d 192.168.201.126@tcp -m GET -e local_dropped -r 1 [ 792.850644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.26@tcp -d 192.168.201.126@tcp1 -m GET -e local_dropped -r 1 [ 792.875638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.26@tcp -d 192.168.201.26@tcp -m GET -e local_dropped -r 1 [ 792.904650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.26@tcp -d 192.168.201.26@tcp1 -m GET -e local_dropped -r 1 [ 792.936645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.26@tcp1 -d 192.168.201.126@tcp -m GET -e local_dropped -r 1 [ 792.964373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.26@tcp1 -d 192.168.201.126@tcp1 -m GET -e local_dropped -r 1 [ 792.993455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.26@tcp1 -d 192.168.201.26@tcp -m GET -e local_dropped -r 1 [ 793.027714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.26@tcp1 -d 192.168.201.26@tcp1 -m GET -e local_dropped -r 1 [ 793.100662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.126@tcp [ 793.116386] LNet: There was an unexpected network error while writing to 192.168.201.126: rc = -22 [ 793.120554] LNet: Skipped 2 previous similar messages [ 793.332155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 793.353699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 793.380249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 793.413993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 793.441923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 793.483233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 793.513633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 793.542409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 793.559334] LNet: 58474:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 793.575213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 793.602531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 793.636212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 793.669050] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 793.704097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 793.735954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 793.764858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 794.224569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.26@tcp -d 192.168.201.126@tcp -m GET -e local_aborted -r 1 [ 794.253181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.26@tcp -d 192.168.201.126@tcp1 -m GET -e local_aborted -r 1 [ 794.292897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.26@tcp -d 192.168.201.26@tcp -m GET -e local_aborted -r 1 [ 794.331069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.26@tcp -d 192.168.201.26@tcp1 -m GET -e local_aborted -r 1 [ 794.368048] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.26@tcp1 -d 192.168.201.126@tcp -m GET -e local_aborted -r 1 [ 794.403895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.26@tcp1 -d 192.168.201.126@tcp1 -m GET -e local_aborted -r 1 [ 794.457387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.26@tcp1 -d 192.168.201.26@tcp -m GET -e local_aborted -r 1 [ 794.521168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.26@tcp1 -d 192.168.201.26@tcp1 -m GET -e local_aborted -r 1 [ 794.566394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.126@tcp [ 794.585315] LNet: There was an unexpected network error while writing to 192.168.201.126: rc = -22 [ 794.589922] LNet: Skipped 2 previous similar messages [ 794.748475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 794.771224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 794.799942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 794.823234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 794.849969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 794.870321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 794.894766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 794.917610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 794.932640] LNet: 58596:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 794.948401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 794.978782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 795.016411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 795.045344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 795.077756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 795.110824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 795.140672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 795.722900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.26@tcp -d 192.168.201.126@tcp -m GET -e local_no_route -r 1 [ 795.757623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.26@tcp -d 192.168.201.126@tcp1 -m GET -e local_no_route -r 1 [ 795.784356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.26@tcp -d 192.168.201.26@tcp -m GET -e local_no_route -r 1 [ 795.815891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.26@tcp -d 192.168.201.26@tcp1 -m GET -e local_no_route -r 1 [ 795.875942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.26@tcp1 -d 192.168.201.126@tcp -m GET -e local_no_route -r 1 [ 795.977693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.26@tcp1 -d 192.168.201.126@tcp1 -m GET -e local_no_route -r 1 [ 796.046425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.26@tcp1 -d 192.168.201.26@tcp -m GET -e local_no_route -r 1 [ 796.111249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.26@tcp1 -d 192.168.201.26@tcp1 -m GET -e local_no_route -r 1 [ 796.183883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.126@tcp [ 796.416612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 796.447869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 796.476462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 796.510948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 796.549844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 796.604458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 796.628350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 796.646986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 796.667593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 796.694416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 796.719476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 796.743919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 796.767831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 796.797175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 796.824914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 797.319040] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.26@tcp -d 192.168.201.126@tcp -m GET -e local_timeout -r 1 [ 797.345768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.26@tcp -d 192.168.201.126@tcp1 -m GET -e local_timeout -r 1 [ 797.381952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.26@tcp -d 192.168.201.26@tcp -m GET -e local_timeout -r 1 [ 797.415922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.26@tcp -d 192.168.201.26@tcp1 -m GET -e local_timeout -r 1 [ 797.450542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.26@tcp1 -d 192.168.201.126@tcp -m GET -e local_timeout -r 1 [ 797.493682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.26@tcp1 -d 192.168.201.126@tcp1 -m GET -e local_timeout -r 1 [ 797.541949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.26@tcp1 -d 192.168.201.26@tcp -m GET -e local_timeout -r 1 [ 797.595901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.26@tcp1 -d 192.168.201.26@tcp1 -m GET -e local_timeout -r 1 [ 797.641979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.126@tcp [ 797.659986] LNet: There was an unexpected network error while writing to 192.168.201.126: rc = -22 [ 797.668104] LNet: Skipped 5 previous similar messages [ 797.920898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 797.955863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 797.994249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 798.017256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 798.044840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 798.072765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 798.121575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 798.148827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 798.172837] LNet: 58838:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 798.176792] LNet: 58838:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 798.192033] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 798.225518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 798.262335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 798.313944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 798.342862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 798.372197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 798.401880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 798.930550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.26@tcp -d 192.168.201.126@tcp -m GET -e local_error -r 1 [ 798.971678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.26@tcp -d 192.168.201.126@tcp1 -m GET -e local_error -r 1 [ 799.021460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.26@tcp -d 192.168.201.26@tcp -m GET -e local_error -r 1 [ 799.063916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.26@tcp -d 192.168.201.26@tcp1 -m GET -e local_error -r 1 [ 799.120062] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.26@tcp1 -d 192.168.201.126@tcp -m GET -e local_error -r 1 [ 799.175060] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.26@tcp1 -d 192.168.201.126@tcp1 -m GET -e local_error -r 1 [ 799.220217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.26@tcp1 -d 192.168.201.26@tcp -m GET -e local_error -r 1 [ 799.276218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.26@tcp1 -d 192.168.201.26@tcp1 -m GET -e local_error -r 1 [ 799.322633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.126@tcp [ 799.568392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 799.590768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 799.613493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 799.637183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 799.666948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 799.699359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 799.736949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 799.789537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 799.822908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 799.852677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 799.901813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 799.931996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 799.968633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 800.004471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 800.039480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 805.434187] LNet: 59036:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 805.444293] LNetError: 59036:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 805.455038] LNet: Removed LNI 192.168.201.26@tcp [ 805.481212] LNet: Removed LNI 192.168.201.26@tcp1 [ 805.899961] Key type .llcrypt unregistered [ 805.901840] Key type ._llcrypt unregistered [ 821.302166] Key type ._llcrypt registered [ 821.303976] Key type .llcrypt registered [ 822.391553] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 12:37:11 (1780331831) [ 823.098376] Key type .llcrypt unregistered [ 823.101740] Key type ._llcrypt unregistered [ 829.475237] Key type ._llcrypt registered [ 829.478399] Key type .llcrypt registered [ 829.698920] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 829.718456] alg: No test for adler32 (adler32-zlib) [ 830.637680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 830.705473] LNet: Added LNI 192.168.201.26@tcp [8/256/0/180] [ 838.458596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.126@tcp [ 839.080973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.26@tcp -d 192.168.201.126@tcp -m GET -e remote_dropped -r 1 [ 839.115514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.26@tcp -d 192.168.201.26@tcp -m GET -e remote_dropped -r 1 [ 839.142851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.126@tcp [ 839.161640] LNet: There was an unexpected network error while writing to 192.168.201.126: rc = -22 [ 839.364248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 839.392723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 839.412608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 839.436232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 839.465896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 839.500670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 839.537936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 839.575766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 839.591251] LNet: 60286:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 839.626136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 839.651536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 839.688449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 839.720210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 839.741546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 839.762725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 839.785379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 840.292714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.26@tcp -d 192.168.201.126@tcp -m GET -e remote_error -r 1 [ 840.329426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.26@tcp -d 192.168.201.26@tcp -m GET -e remote_error -r 1 [ 840.363629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.126@tcp [ 840.377476] LNet: There was an unexpected network error while writing to 192.168.201.126: rc = -22 [ 840.571502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 840.600342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 840.632439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 840.673406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 840.696774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 840.712331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 840.728744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 840.742683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 840.751296] LNet: 60396:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 840.760886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 840.782446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 840.801428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 840.840039] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 840.868897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 840.902586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 840.936869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 841.448413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.26@tcp -d 192.168.201.126@tcp -m GET -e remote_timeout -r 1 [ 841.472480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.26@tcp -d 192.168.201.26@tcp -m GET -e remote_timeout -r 1 [ 841.490960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.126@tcp [ 841.503051] LNet: There was an unexpected network error while writing to 192.168.201.126: rc = -22 [ 841.640122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 841.657690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 841.675486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 841.696972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 841.720180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 841.748374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 841.765248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 841.799714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 841.816452] LNet: 60505:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 841.836663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 841.855440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 841.886594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 841.910196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 841.925968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 841.942313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 841.973431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 842.219868] LNet: 60546:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 842.224538] LNetError: 60546:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 842.234921] LNet: Removed LNI 192.168.201.26@tcp [ 842.514504] Key type .llcrypt unregistered [ 842.515910] Key type ._llcrypt unregistered [ 855.958633] Key type ._llcrypt registered [ 855.959934] Key type .llcrypt registered [ 857.286580] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 12:37:46 (1780331866) [ 858.189550] Key type .llcrypt unregistered [ 858.191383] Key type ._llcrypt unregistered [ 864.072235] Key type ._llcrypt registered [ 864.075468] Key type .llcrypt registered [ 864.463932] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 864.485329] alg: No test for adler32 (adler32-zlib) [ 865.386615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 865.411606] LNet: Added LNI 192.168.201.26@tcp [8/256/0/180] [ 873.436811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.126@tcp [ 885.342431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 885.397542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 885.435269] LNet: Added LNI 192.168.201.26@tcp1 [8/256/0/180] [ 886.773447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.26@tcp -d 192.168.201.126@tcp -m GET -e remote_dropped -r 1 [ 886.793457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.26@tcp -d 192.168.201.126@tcp1 -m GET -e remote_dropped -r 1 [ 886.841873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.26@tcp -d 192.168.201.26@tcp -m GET -e remote_dropped -r 1 [ 886.903974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.26@tcp -d 192.168.201.26@tcp1 -m GET -e remote_dropped -r 1 [ 886.972853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.26@tcp1 -d 192.168.201.126@tcp -m GET -e remote_dropped -r 1 [ 887.052874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.26@tcp1 -d 192.168.201.126@tcp1 -m GET -e remote_dropped -r 1 [ 887.101359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.26@tcp1 -d 192.168.201.26@tcp -m GET -e remote_dropped -r 1 [ 887.152817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.26@tcp1 -d 192.168.201.26@tcp1 -m GET -e remote_dropped -r 1 [ 887.196822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.126@tcp [ 887.215551] LNet: There was an unexpected network error while writing to 192.168.201.126: rc = -22 [ 887.220141] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.126@tcp [ 887.443984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 887.466318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 887.490910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 887.518995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 887.561748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 887.587778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 887.622232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 887.671852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 887.691213] LNet: 62010:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 887.709857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 887.748964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 887.785896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 887.839645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 887.895806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 887.948909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 887.992734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 888.028054] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 888.459661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.26@tcp -d 192.168.201.126@tcp -m GET -e remote_error -r 1 [ 888.492966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.26@tcp -d 192.168.201.126@tcp1 -m GET -e remote_error -r 1 [ 888.525526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.26@tcp -d 192.168.201.26@tcp -m GET -e remote_error -r 1 [ 888.549661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.26@tcp -d 192.168.201.26@tcp1 -m GET -e remote_error -r 1 [ 888.585939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.26@tcp1 -d 192.168.201.126@tcp -m GET -e remote_error -r 1 [ 888.616600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.26@tcp1 -d 192.168.201.126@tcp1 -m GET -e remote_error -r 1 [ 888.670328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.26@tcp1 -d 192.168.201.26@tcp -m GET -e remote_error -r 1 [ 888.711317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.26@tcp1 -d 192.168.201.26@tcp1 -m GET -e remote_error -r 1 [ 888.792326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.126@tcp [ 888.820392] LNet: There was an unexpected network error while writing to 192.168.201.126: rc = -22 [ 888.832685] LNet: Skipped 2 previous similar messages [ 889.096912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 889.137549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 889.163707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 889.213444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 889.300059] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 889.376937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 889.442984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 889.481431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 889.524797] LNet: 62133:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 889.537853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 889.586735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 889.621819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 889.646474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 889.670650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 889.701635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 889.730959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 889.756925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 890.235678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.26@tcp -d 192.168.201.126@tcp -m GET -e remote_timeout -r 1 [ 890.276414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.26@tcp -d 192.168.201.126@tcp1 -m GET -e remote_timeout -r 1 [ 890.308667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.26@tcp -d 192.168.201.26@tcp -m GET -e remote_timeout -r 1 [ 890.338810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.26@tcp -d 192.168.201.26@tcp1 -m GET -e remote_timeout -r 1 [ 890.420308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.26@tcp1 -d 192.168.201.126@tcp -m GET -e remote_timeout -r 1 [ 890.462764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.26@tcp1 -d 192.168.201.126@tcp1 -m GET -e remote_timeout -r 1 [ 890.487655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.26@tcp1 -d 192.168.201.26@tcp -m GET -e remote_timeout -r 1 [ 890.516889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.26@tcp1 -d 192.168.201.26@tcp1 -m GET -e remote_timeout -r 1 [ 890.548367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.126@tcp [ 890.567504] LNet: There was an unexpected network error while writing to 192.168.201.126: rc = -22 [ 890.572176] LNet: Skipped 1 previous similar message [ 890.810349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 890.851390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 890.885106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 890.936433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 890.969533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 890.994813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 891.023458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 891.056365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 891.078389] LNet: 62257:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 891.098487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 891.138974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 891.183674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 891.218658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 891.260798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 891.297474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 891.325475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 891.358484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 896.354914] LNet: 62335:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 896.360123] LNetError: 62335:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 896.367401] LNet: Removed LNI 192.168.201.26@tcp [ 896.381700] LNet: Removed LNI 192.168.201.26@tcp1 [ 896.587551] Key type .llcrypt unregistered [ 896.589734] Key type ._llcrypt unregistered [ 912.945393] Key type ._llcrypt registered [ 912.947455] Key type .llcrypt registered [ 914.252885] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 12:38:43 (1780331923) [ 915.946486] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 918.122953] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 12:38:47 (1780331927) [ 918.914600] Key type .llcrypt unregistered [ 918.917389] Key type ._llcrypt unregistered [ 925.626561] Key type ._llcrypt registered [ 925.634943] Key type .llcrypt registered [ 926.015051] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 926.040693] alg: No test for adler32 (adler32-zlib) [ 927.032953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 927.071479] LNet: Added LNI 192.168.201.26@tcp [8/256/0/180] [ 935.003964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.126@tcp [ 935.686758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.26@tcp -d 192.168.201.126@tcp -m GET -e network_timeout -r 1 [ 935.718347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.26@tcp -d 192.168.201.26@tcp -m GET -e network_timeout -r 1 [ 935.743274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.126@tcp [ 935.775626] LNet: There was an unexpected network error while writing to 192.168.201.126: rc = -22 [ 936.010402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 936.065735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 936.105384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 936.156508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 936.202225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 936.236224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 936.276965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 936.291898] LNet: 1 local NIs in recovery (showing 1): 192.168.201.26@tcp [ 936.301512] LNet: There was an unexpected network error while writing to 192.168.201.26: rc = -22 [ 936.317094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 936.334686] LNet: 63881:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 936.361233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 936.396738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 936.431750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 936.472677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 936.536506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 936.583165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 936.905457] LNet: 63920:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 936.915582] LNetError: 63920:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 936.935493] LNet: Removed LNI 192.168.201.26@tcp [ 937.370710] Key type .llcrypt unregistered [ 937.372531] Key type ._llcrypt unregistered [ 954.970743] Key type ._llcrypt registered [ 954.972713] Key type .llcrypt registered [ 955.121924] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 955.137522] alg: No test for adler32 (adler32-zlib) [ 956.143585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 956.190890] LNet: Added LNI 192.168.201.26@tcp [8/256/0/180] [ 964.270445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.126@tcp [ 973.776237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 973.821116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 973.846873] LNet: Added LNI 192.168.201.26@tcp1 [8/256/0/180] [ 975.712759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.26@tcp -d 192.168.201.126@tcp -m GET -e network_timeout -r 1 [ 975.769935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.26@tcp -d 192.168.201.126@tcp1 -m GET -e network_timeout -r 1 [ 975.803538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.26@tcp -d 192.168.201.26@tcp -m GET -e network_timeout -r 1 [ 975.845511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.26@tcp -d 192.168.201.26@tcp1 -m GET -e network_timeout -r 1 [ 975.888788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.26@tcp1 -d 192.168.201.126@tcp -m GET -e network_timeout -r 1 [ 975.944647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.26@tcp1 -d 192.168.201.126@tcp1 -m GET -e network_timeout -r 1 [ 976.010699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.26@tcp1 -d 192.168.201.26@tcp -m GET -e network_timeout -r 1 [ 976.055970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.26@tcp1 -d 192.168.201.26@tcp1 -m GET -e network_timeout -r 1 [ 976.103274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.126@tcp [ 976.130079] LNet: There was an unexpected network error while writing to 192.168.201.126: rc = -22 [ 976.390096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 976.431702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 976.457653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 976.491387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 976.531941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 976.560582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 976.591092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 976.626515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 976.643394] LNet: 64799:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 976.670402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 976.677142] LNet: 1 local NIs in recovery (showing 1): 192.168.201.26@tcp [ 976.685083] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.126@tcp [ 976.707042] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 976.729964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 976.771208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 976.808205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 976.859449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 981.925667] LNet: There was an unexpected network error while writing to 192.168.201.126: rc = -22 [ 982.149366] LNet: 64873:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 982.158253] LNetError: 64873:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 982.166816] LNet: Removed LNI 192.168.201.26@tcp [ 982.204572] LNet: Removed LNI 192.168.201.26@tcp1 [ 982.566656] Key type .llcrypt unregistered [ 982.569235] Key type ._llcrypt unregistered [ 1002.499888] Key type ._llcrypt registered [ 1002.501510] Key type .llcrypt registered [ 1004.203261] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 12:40:13 (1780332013) [ 1004.681245] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1004.702797] alg: No test for adler32 (adler32-zlib) [ 1005.647554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1005.724971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1005.748559] LNet: Added LNI 192.168.201.26@tcp [8/256/0/180] [ 1006.609755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1006.626022] LNet: Added LNI 192.168.201.26@tcp1 [8/256/0/180] [ 1007.195977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.26@tcp [ 1007.298850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1007.359995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1007.418593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1007.456834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1007.478468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1007.502935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.201.26@tcp [ 1007.518746] LNetError: 65973:0:(api-ni.c:4350:lnet_ni_set_healthv()) manually adding local NI 192.168.201.26@tcp to recovery [ 1007.775510] LNet: 1 local NIs in recovery (showing 1): 192.168.201.26@tcp [ 1007.777334] LNet: There was an unexpected network error while writing to 192.168.201.26: rc = -22 [ 1009.824092] LNet: There was an unexpected network error while writing to 192.168.201.26: rc = -22 [ 1009.963655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1013.919366] LNet: There was an unexpected network error while writing to 192.168.201.26: rc = -22 [ 1014.603265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1014.632602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1014.663676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1014.676162] LNet: 66080:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1014.680951] LNetError: 66080:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1015.711211] LNet: Removed LNI 192.168.201.26@tcp [ 1015.715981] LNetError: 66080:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1015.730882] LNet: Removed LNI 192.168.201.26@tcp1 [ 1015.755492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1015.809563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1015.831787] LNet: Added LNI 192.168.201.26@tcp [8/256/0/180] [ 1016.567569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1016.594745] LNet: Added LNI 192.168.201.26@tcp1 [8/256/0/180] [ 1017.467682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.26@tcp [ 1017.512526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1017.558435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1017.593259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1017.622604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1017.661492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1017.687414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1017.707342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.201.26@tcp [ 1017.720204] LNetError: 66352:0:(api-ni.c:4350:lnet_ni_set_healthv()) manually adding local NI 192.168.201.26@tcp to recovery [ 1017.827744] LNet: There was an unexpected network error while writing to 192.168.201.26: rc = -22 [ 1020.171322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1023.967615] LNet: There was an unexpected network error while writing to 192.168.201.26: rc = -22 [ 1023.980593] LNet: Skipped 1 previous similar message [ 1028.267400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1028.318981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1028.380342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1035.597938] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 12:40:45 (1780332045) [ 1036.014763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1036.060366] LNet: 67062:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1036.066061] LNetError: 67062:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1037.087227] LNet: Removed LNI 192.168.201.26@tcp [ 1037.099311] LNet: Removed LNI 192.168.201.26@tcp1 [ 1037.167879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1037.252880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1037.282552] LNet: Added LNI 192.168.201.26@tcp [8/256/0/180] [ 1038.265435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1038.292465] LNet: Added LNI 192.168.201.26@tcp1 [8/256/0/180] [ 1039.101995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.26@tcp [ 1039.157309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1039.180238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1039.212208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1039.237791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1039.269569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1039.299936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.201.26@tcp --health 0 [ 1039.418486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1040.287519] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.26@tcp [ 1040.288308] LNet: There was an unexpected network error while writing to 192.168.201.26: rc = -22 [ 1040.305415] LNet: Skipped 1 previous similar message [ 1054.822907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1055.004713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1055.066400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1055.157857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.201.26@tcp --health 500 [ 1055.308274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1062.297329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1062.378271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1062.422719] LNet: 67555:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1062.441953] LNetError: 67555:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1062.455521] LNetError: 67555:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1062.467638] LNet: Removed LNI 192.168.201.26@tcp [ 1062.491455] LNet: Removed LNI 192.168.201.26@tcp1 [ 1062.579617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1062.700846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1062.749266] LNet: Added LNI 192.168.201.26@tcp [8/256/0/180] [ 1063.539170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1063.553294] LNet: Added LNI 192.168.201.26@tcp1 [8/256/0/180] [ 1064.388490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.26@tcp [ 1064.475252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1064.566025] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1064.589640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1064.604982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1064.630799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1064.661410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1064.707858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.201.26@tcp --health 0 [ 1065.761433] LNet: There was an unexpected network error while writing to 192.168.201.26: rc = -22 [ 1065.770586] LNet: Skipped 3 previous similar messages [ 1066.218876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1076.592577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1076.615799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1076.640880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1082.961641] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 12:41:32 (1780332092) [ 1083.323377] LNet: 68600:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1083.326718] LNetError: 68600:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1083.329118] LNetError: 68600:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1083.331468] LNet: Removed LNI 192.168.201.26@tcp [ 1083.341314] LNet: Removed LNI 192.168.201.26@tcp1 [ 1083.723495] Key type .llcrypt unregistered [ 1083.729497] Key type ._llcrypt unregistered [ 1091.587118] Key type ._llcrypt registered [ 1091.588803] Key type .llcrypt registered [ 1091.817902] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1091.841350] alg: No test for adler32 (adler32-zlib) [ 1092.836821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1092.864647] LNet: Added LNI 192.168.201.26@tcp [8/256/0/180] [ 1101.949202] Lustre: DEBUG MARKER: Initial discovery [ 1102.056310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.201.126@tcp [ 1104.484701] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1104.547645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1104.599797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.201.126@tcp [ 1104.624197] LNet: There was an unexpected network error while writing to 192.168.201.126: rc = -22 [ 1104.645854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1104.683201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.201.126@tcp -r 1 -m GET -l 3 [ 1104.738797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.201.126@tcp -r 1 -m GET -e local_error [ 1104.777556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.201.126@tcp -r 1 -m PUT -l 6 [ 1105.184922] LNet: 1 local NIs in recovery (showing 1): 192.168.201.26@tcp [ 1106.349308] Lustre: DEBUG MARKER: Force oleg126-client.virtnet to discover 192.168.201.126@tcp (in background) [ 1106.457203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.201.126@tcp [ 1107.816121] Lustre: DEBUG MARKER: Force oleg126-server to discover 192.168.201.26@tcp [ 1109.540550] LNet: There was an unexpected network error while writing to 192.168.201.126: rc = -22 [ 1110.249479] Lustre: DEBUG MARKER: Wait for 69353 [ 1111.671153] Lustre: DEBUG MARKER: Finished wait on 69353 [ 1111.797481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay del -a [ 1111.847368] LNet: There was an unexpected network error while writing to 192.168.201.126: rc = -22 [ 1111.851581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1112.124508] LNet: 69811:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1112.129075] LNetError: 69811:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1113.194676] LNet: Removed LNI 192.168.201.26@tcp [ 1113.617571] Key type .llcrypt unregistered [ 1113.619765] Key type ._llcrypt unregistered [ 1131.579465] Key type ._llcrypt registered [ 1131.581544] Key type .llcrypt registered [ 1132.861753] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 12:42:22 (1780332142) [ 1133.364501] Key type .llcrypt unregistered [ 1133.366954] Key type ._llcrypt unregistered [ 1139.799348] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1140.041193] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1140.061706] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1140.064103] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1142.735533] Key type ._llcrypt registered [ 1142.741784] Key type .llcrypt registered [ 1142.985130] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1142.996771] alg: No test for adler32 (adler32-zlib) [ 1144.010739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1144.118381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1144.154111] LNet: Added LNI 192.168.201.26@tcp [8/256/0/180] [ 1145.151841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1145.181157] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1155.075620] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 12:42:44 (1780332164) [ 1155.320685] LNet: 71890:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1155.327373] LNetError: 71890:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1155.330828] LNet: Removed LNI 192.168.201.26@tcp [ 1155.345039] LNet: Removed LNI 10.1.2.3@tcp [ 1156.218411] Key type .llcrypt unregistered [ 1156.220717] Key type ._llcrypt unregistered [ 1164.690505] Key type ._llcrypt registered [ 1164.692016] Key type .llcrypt registered [ 1164.972400] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1164.993503] alg: No test for adler32 (adler32-zlib) [ 1165.980382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1166.038751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1166.056065] LNet: Added LNI 192.168.201.26@tcp [8/256/0/180] [ 1167.139320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1167.160599] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1168.335614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.201.26@tcp [ 1169.375242] LNetError: Unexpected error -22 connecting to 192.168.201.26@tcp at host 192.168.201.26:988 [ 1175.460661] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 12:43:04 (1780332184) [ 1175.711354] LNet: 73023:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1175.720957] LNetError: 73023:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1175.726474] LNet: Removed LNI 192.168.201.26@tcp [ 1175.736420] LNet: Removed LNI 10.1.2.3@tcp [ 1176.367048] Key type .llcrypt unregistered [ 1176.368943] Key type ._llcrypt unregistered [ 1183.051454] Key type ._llcrypt registered [ 1183.057212] Key type .llcrypt registered [ 1183.364075] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1183.388664] alg: No test for adler32 (adler32-zlib) [ 1184.372221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1184.449592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1184.489320] LNet: Added LNI 192.168.201.26@tcp1 [8/256/0/180] [ 1185.142793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1185.167697] LNet: Added LNI 192.168.201.26@tcp2 [8/256/0/180] [ 1186.280726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.201.26@tcp1 --nid 192.168.201.26@tcp2 [ 1191.801596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.26@tcp1 192.168.201.26@tcp2 [ 1191.825570] LNetError: 73858:0:(lib-move.c:2007:lnet_handle_find_routed_path()) Unable to send message from 192.168.201.26@tcp1 to 192.168.201.26@tcp2 - Route table may be misconfigured [ 1191.858407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.26@tcp2 192.168.201.26@tcp1 [ 1198.455025] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 12:43:27 (1780332207) [ 1198.621067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1198.657226] LNet: 74428:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1198.679702] LNetError: 74428:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1199.712156] LNet: Removed LNI 192.168.201.26@tcp1 [ 1199.714058] LNetError: 74428:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1199.735596] LNet: Removed LNI 192.168.201.26@tcp2 [ 1199.765512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1199.864227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1199.921413] LNet: Added LNI 192.168.201.26@tcp [8/256/0/180] [ 1200.761359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1200.791914] LNet: Added LNI 192.168.201.26@tcp1 [8/256/0/180] [ 1201.881162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.26@tcp [ 1201.957873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.26@tcp 192.168.201.26@tcp [ 1202.001234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.26@tcp1 192.168.201.26@tcp1 [ 1202.058271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.201.26@tcp -d 192.168.201.26@tcp -e network_timeout [ 1202.147330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.201.26@tcp -d 192.168.201.26@tcp1 -e network_timeout [ 1202.218375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.201.26@tcp1 -d 192.168.201.26@tcp -e network_timeout [ 1202.256308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.201.26@tcp1 -d 192.168.201.26@tcp1 -e network_timeout [ 1202.300269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.26@tcp [ 1202.333697] LNet: There was an unexpected network error while writing to 192.168.201.26: rc = -22 [ 1202.357356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1202.505974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1202.595700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1202.913847] LNet: 1 local NIs in recovery (showing 1): 192.168.201.26@tcp [ 1209.455181] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 12:43:38 (1780332218) [ 1209.598853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1209.640688] LNet: 75287:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1209.654909] LNetError: 75287:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1209.666194] LNet: Removed LNI 192.168.201.26@tcp [ 1210.728881] LNet: Removed LNI 192.168.201.26@tcp1 [ 1210.756659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1210.860248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1211.133514] LNet: 75330:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1211.666727] Key type .llcrypt unregistered [ 1211.668974] Key type ._llcrypt unregistered [ 1223.750719] Key type ._llcrypt registered [ 1223.754939] Key type .llcrypt registered [ 1224.985338] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 12:43:54 (1780332234) [ 1225.265880] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1225.279689] alg: No test for adler32 (adler32-zlib) [ 1226.177536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1227.402546] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1228.837454] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 12:43:58 (1780332238) [ 1229.039638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1229.090564] LNet: 76428:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1229.114378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1229.176841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1229.215600] LNet: Added LNI 192.168.201.26@tcp [8/256/0/180] [ 1229.933362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1229.952827] LNet: Added LNI 192.168.201.26@tcp1 [8/256/0/180] [ 1230.804923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.26@tcp [ 1230.839972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.26@tcp1 [ 1230.886604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.26@tcp1 [ 1237.605811] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 12:44:06 (1780332246) [ 1239.421400] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg126-server [ 1241.151729] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 12:44:10 (1780332250) [ 1242.615311] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg126-server [ 1244.401097] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 12:44:13 (1780332253) [ 1246.421925] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg126-server [ 1248.440042] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 12:44:17 (1780332257) [ 1250.630822] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg126-server [ 1252.513592] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 12:44:21 (1780332261) [ 1254.536552] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg126-server [ 1256.602605] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 12:44:25 (1780332265) [ 1258.761445] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg126-server [ 1261.237797] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 12:44:30 (1780332270) [ 1263.480947] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg126-server [ 1265.512817] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 12:44:34 (1780332274) [ 1267.486756] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg126-server [ 1269.445156] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 12:44:38 (1780332278) [ 1271.512441] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg126-server [ 1273.609157] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 12:44:42 (1780332282) [ 1273.751598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1273.785837] LNet: 80299:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1273.798306] LNetError: 80299:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1273.806173] LNet: Removed LNI 192.168.201.26@tcp [ 1274.854177] LNet: Removed LNI 192.168.201.26@tcp1 [ 1274.887587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1274.961041] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1275.009396] LNet: Added LNI 192.168.201.26@tcp [8/256/0/180] [ 1276.091173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1276.237307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.26@tcp [ 1276.459093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1276.480937] LNet: 80454:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1276.489910] LNetError: 80454:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1276.498509] LNetError: 80454:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1276.522924] LNet: Removed LNI 192.168.201.26@tcp [ 1276.561163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1276.635861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1276.672921] LNet: Added LNI 192.168.201.26@tcp [8/256/0/180] [ 1277.540612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.26@tcp [ 1277.758909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1277.801258] LNetError: 80599:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1278.822758] LNet: Removed LNI 192.168.201.26@tcp [ 1278.860400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1278.986766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1279.043299] LNet: Added LNI 192.168.201.26@tcp [8/256/0/180] [ 1279.904472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1279.960553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.26@tcp [ 1280.092182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1280.116713] LNet: 80746:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1280.121403] LNet: 80746:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1280.126516] LNetError: 80746:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1281.189943] LNet: Removed LNI 192.168.201.26@tcp [ 1281.234819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1281.363691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1281.424558] LNet: Added LNI 192.168.201.26@tcp [8/256/0/180] [ 1282.513915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.26@tcp [ 1282.778941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1283.879429] LNet: Removed LNI 192.168.201.26@tcp [ 1283.913655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1284.019860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1284.064567] LNet: Added LNI 192.168.201.26@tcp [8/256/0/180] [ 1285.045668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1285.154636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.26@tcp [ 1285.447682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1285.481574] LNet: 81039:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1285.488278] LNet: 81039:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1285.493664] LNetError: 81039:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1285.498764] LNetError: 81039:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1285.514695] LNet: Removed LNI 192.168.201.26@tcp [ 1285.587130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1285.673841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1285.727861] LNet: Added LNI 192.168.201.26@tcp [8/256/0/180] [ 1286.548621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.26@tcp [ 1286.750610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1287.799887] LNet: Removed LNI 192.168.201.26@tcp [ 1287.862074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1287.967753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1287.989405] LNet: Added LNI 192.168.201.26@tcp [8/256/0/180] [ 1288.644966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1288.734427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.26@tcp [ 1289.066623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1290.176431] LNet: Removed LNI 192.168.201.26@tcp [ 1290.216070] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1290.313226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1290.330162] LNet: Added LNI 192.168.201.26@tcp [8/256/0/180] [ 1291.170162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.26@tcp [ 1291.427214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1292.520180] LNet: Removed LNI 192.168.201.26@tcp [ 1292.565989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1292.647974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1292.681658] LNet: Added LNI 192.168.201.26@tcp [8/256/0/180] [ 1293.442345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1293.528722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.26@tcp [ 1293.759397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1293.779466] LNet: 81624:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1293.784284] LNet: 81624:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 1293.788828] LNetError: 81624:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1293.791716] LNetError: 81624:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 1294.821531] LNet: Removed LNI 192.168.201.26@tcp [ 1294.869664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1294.974205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1295.012751] LNet: Added LNI 192.168.201.26@tcp [8/256/0/180] [ 1295.740551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.26@tcp [ 1295.933555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1297.000654] LNet: Removed LNI 192.168.201.26@tcp [ 1297.056895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1297.136940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1297.191793] LNet: Added LNI 192.168.201.26@tcp [8/256/0/180] [ 1297.850716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1297.932276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.26@tcp [ 1298.171526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1299.245230] LNet: Removed LNI 192.168.201.26@tcp [ 1299.304844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1299.356320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1299.382240] LNet: Added LNI 192.168.201.26@tcp [8/256/0/180] [ 1299.984315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.26@tcp [ 1300.177420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1301.244239] LNet: Removed LNI 192.168.201.26@tcp [ 1301.303095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1301.370590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1301.406416] LNet: Added LNI 192.168.201.26@tcp [8/256/0/180] [ 1302.293835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1302.377464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1302.398950] LNet: Removed LNI 192.168.201.26@tcp [ 1302.413649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1302.468634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1302.486638] LNet: Added LNI 192.168.201.26@tcp [8/256/0/180] [ 1303.192749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1310.749457] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 12:45:20 (1780332320) [ 1310.884634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1310.904861] LNet: 82892:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1310.914050] LNet: 82892:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 1310.928457] LNetError: 82892:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1310.946471] LNetError: 82892:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 1310.969167] LNet: Removed LNI 192.168.201.26@tcp [ 1310.999854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1311.095564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1311.121763] LNet: Added LNI 192.168.201.26@tcp231 [8/256/0/180] [ 1312.191516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1312.242208] LNet: Removed LNI 192.168.201.26@tcp231 [ 1312.296898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1312.438256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1312.479176] LNet: Added LNI 192.168.201.26@tcp231 [8/256/0/180] [ 1312.483147] LNet: 83046:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1313.040420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1313.096197] LNet: Removed LNI 192.168.201.26@tcp231 [ 1313.117643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1313.137875] LNet: Added LNI 192.168.201.26@tcp231 [8/256/0/180] [ 1314.382560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1314.414800] LNet: Removed LNI 192.168.201.26@tcp231 [ 1314.429910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1314.521380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1314.585257] LNet: Added LNI 192.168.201.26@tcp231 [8/256/0/0] [ 1314.599569] LNet: 83198:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1315.184474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1315.221272] LNet: Removed LNI 192.168.201.26@tcp231 [ 1315.252886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1315.282725] LNet: Added LNI 192.168.201.26@tcp231 [8/256/0/0] [ 1316.671454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1316.709930] LNet: Removed LNI 192.168.201.26@tcp231 [ 1316.735290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1316.796288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1316.810527] LNet: Added LNI 192.168.201.26@tcp231 [8/256/0/60] [ 1316.813475] LNet: 83350:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1317.444360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1317.474480] LNet: Removed LNI 192.168.201.26@tcp231 [ 1317.503214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1317.542782] LNet: Added LNI 192.168.201.26@tcp231 [8/256/0/60] [ 1319.014958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1319.072530] LNet: Removed LNI 192.168.201.26@tcp231 [ 1319.104271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1319.204599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1319.243625] LNet: Added LNI 192.168.201.26@tcp231 [8/256/0/180] [ 1319.249517] LNet: 83501:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1320.008843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1320.047263] LNet: Removed LNI 192.168.201.26@tcp231 [ 1320.061853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1320.090488] LNet: Added LNI 192.168.201.26@tcp231 [8/256/0/180] [ 1327.897336] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 12:45:37 (1780332337) [ 1328.049262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1328.112069] LNet: Removed LNI 192.168.201.26@tcp231 [ 1328.129330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1328.188781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1328.206465] LNet: Added LNI 192.168.201.26@tcp232 [8/256/0/180] [ 1328.799460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1328.839779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1328.869396] LNet: Removed LNI 192.168.201.26@tcp232 [ 1328.882288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1328.898338] LNet: Added LNI 192.168.201.26@tcp232 [8/256/0/180] [ 1329.331217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1329.361953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1329.400299] LNet: Removed LNI 192.168.201.26@tcp232 [ 1329.427112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1329.456726] LNet: Added LNI 192.168.201.26@tcp232 [8/256/0/180] [ 1330.051237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1330.083658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1330.111600] LNet: Removed LNI 192.168.201.26@tcp232 [ 1330.123803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1330.139964] LNet: Added LNI 192.168.201.26@tcp232 [8/256/0/180] [ 1330.805987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1330.853992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1330.896152] LNet: Removed LNI 192.168.201.26@tcp232 [ 1330.919087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1336.908593] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 12:45:46 (1780332346) [ 1338.070702] Key type .llcrypt unregistered [ 1338.076447] Key type ._llcrypt unregistered [ 1344.310530] Key type ._llcrypt registered [ 1344.315229] Key type .llcrypt registered [ 1344.651959] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1344.678307] alg: No test for adler32 (adler32-zlib) [ 1345.662346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1345.696767] LNet: Added LNI 192.168.201.26@tcp [8/256/0/180] [ 1354.259762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.126@tcp [ 1363.935250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1363.976766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1364.014623] LNet: Added LNI 192.168.201.26@tcp1 [8/256/0/180] [ 1365.358357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.26@tcp -d 192.168.201.126@tcp -m GET -e local_interrupt -r 2 [ 1365.404968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.26@tcp -d 192.168.201.126@tcp1 -m GET -e local_interrupt -r 2 [ 1365.454569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.26@tcp -d 192.168.201.26@tcp -m GET -e local_interrupt -r 2 [ 1365.481259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.26@tcp -d 192.168.201.26@tcp1 -m GET -e local_interrupt -r 2 [ 1365.510281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.26@tcp1 -d 192.168.201.126@tcp -m GET -e local_interrupt -r 2 [ 1365.539394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.26@tcp1 -d 192.168.201.126@tcp1 -m GET -e local_interrupt -r 2 [ 1365.566273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.26@tcp1 -d 192.168.201.26@tcp -m GET -e local_interrupt -r 2 [ 1365.605695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.26@tcp1 -d 192.168.201.26@tcp1 -m GET -e local_interrupt -r 2 [ 1365.644902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.26@tcp 192.168.201.126@tcp [ 1365.666464] LNet: There was an unexpected network error while writing to 192.168.201.126: rc = -22 [ 1365.671891] LNet: 1 local NIs in recovery (showing 1): 192.168.201.26@tcp [ 1365.928732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1365.966614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1365.992415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1366.019334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1366.063333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1366.092940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1366.116329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1366.143712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1366.190250] LNet: 86003:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1366.207235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1366.238381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1366.266891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1366.294657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1366.321644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1366.348927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1366.906529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.26@tcp -d 192.168.201.126@tcp -m GET -e local_dropped -r 2 [ 1366.937325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.26@tcp -d 192.168.201.126@tcp1 -m GET -e local_dropped -r 2 [ 1366.970748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.26@tcp -d 192.168.201.26@tcp -m GET -e local_dropped -r 2 [ 1367.008864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.26@tcp -d 192.168.201.26@tcp1 -m GET -e local_dropped -r 2 [ 1367.043996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.26@tcp1 -d 192.168.201.126@tcp -m GET -e local_dropped -r 2 [ 1367.086662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.26@tcp1 -d 192.168.201.126@tcp1 -m GET -e local_dropped -r 2 [ 1367.120469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.26@tcp1 -d 192.168.201.26@tcp -m GET -e local_dropped -r 2 [ 1367.161876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.26@tcp1 -d 192.168.201.26@tcp1 -m GET -e local_dropped -r 2 [ 1367.200583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.26@tcp 192.168.201.126@tcp [ 1367.221249] LNet: There was an unexpected network error while writing to 192.168.201.126: rc = -22 [ 1367.453222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1367.478363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1367.501056] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1367.525216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1367.557550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1367.584272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1367.618148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1367.648771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1367.665765] LNet: 86123:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1367.691949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1367.728537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1367.767554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1367.795453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1367.823658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1367.851949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1368.374958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.26@tcp -d 192.168.201.126@tcp -m GET -e local_aborted -r 2 [ 1368.420403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.26@tcp -d 192.168.201.126@tcp1 -m GET -e local_aborted -r 2 [ 1368.460478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.26@tcp -d 192.168.201.26@tcp -m GET -e local_aborted -r 2 [ 1368.510996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.26@tcp -d 192.168.201.26@tcp1 -m GET -e local_aborted -r 2 [ 1368.544960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.26@tcp1 -d 192.168.201.126@tcp -m GET -e local_aborted -r 2 [ 1368.582399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.26@tcp1 -d 192.168.201.126@tcp1 -m GET -e local_aborted -r 2 [ 1368.632885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.26@tcp1 -d 192.168.201.26@tcp -m GET -e local_aborted -r 2 [ 1368.684790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.26@tcp1 -d 192.168.201.26@tcp1 -m GET -e local_aborted -r 2 [ 1368.734618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.26@tcp 192.168.201.126@tcp [ 1368.758842] LNet: There was an unexpected network error while writing to 192.168.201.126: rc = -22 [ 1368.960372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1368.986762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1369.014946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1369.055346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1369.086804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1369.111370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1369.138473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1369.165876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1369.181219] LNet: 86243:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1369.207286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1369.265348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1369.304464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1369.322817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1369.362171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1369.385291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1369.880761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.26@tcp -d 192.168.201.126@tcp -m GET -e local_no_route -r 2 [ 1369.911367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.26@tcp -d 192.168.201.126@tcp1 -m GET -e local_no_route -r 2 [ 1369.935047] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.26@tcp -d 192.168.201.26@tcp -m GET -e local_no_route -r 2 [ 1369.953373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.26@tcp -d 192.168.201.26@tcp1 -m GET -e local_no_route -r 2 [ 1369.971298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.26@tcp1 -d 192.168.201.126@tcp -m GET -e local_no_route -r 2 [ 1369.992354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.26@tcp1 -d 192.168.201.126@tcp1 -m GET -e local_no_route -r 2 [ 1370.015462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.26@tcp1 -d 192.168.201.26@tcp -m GET -e local_no_route -r 2 [ 1370.088880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.26@tcp1 -d 192.168.201.26@tcp1 -m GET -e local_no_route -r 2 [ 1370.145089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.26@tcp 192.168.201.126@tcp [ 1370.432904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1370.469084] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1370.494968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1370.512813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1370.540511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1370.567284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1370.589393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1370.607872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1370.624570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1370.643711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1370.665574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1370.689966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1370.721966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1370.739314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1371.146569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.26@tcp -d 192.168.201.126@tcp -m GET -e local_timeout -r 2 [ 1371.170958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.26@tcp -d 192.168.201.126@tcp1 -m GET -e local_timeout -r 2 [ 1371.212120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.26@tcp -d 192.168.201.26@tcp -m GET -e local_timeout -r 2 [ 1371.249453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.26@tcp -d 192.168.201.26@tcp1 -m GET -e local_timeout -r 2 [ 1371.271263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.26@tcp1 -d 192.168.201.126@tcp -m GET -e local_timeout -r 2 [ 1371.297950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.26@tcp1 -d 192.168.201.126@tcp1 -m GET -e local_timeout -r 2 [ 1371.331860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.26@tcp1 -d 192.168.201.26@tcp -m GET -e local_timeout -r 2 [ 1371.365262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.26@tcp1 -d 192.168.201.26@tcp1 -m GET -e local_timeout -r 2 [ 1371.391976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.26@tcp 192.168.201.126@tcp [ 1371.406707] LNet: There was an unexpected network error while writing to 192.168.201.126: rc = -22 [ 1371.411375] LNet: Skipped 1 previous similar message [ 1371.650092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1371.682469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1371.719647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1371.758386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1371.794704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1371.843184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1371.867229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1371.896560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1371.914840] LNet: 86484:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1371.919967] LNet: 86484:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 1371.937721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1371.972489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1372.002242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1372.045418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1372.087483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1372.114188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1372.564622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.26@tcp -d 192.168.201.126@tcp -m GET -e remote_dropped -r 2 [ 1372.594607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.26@tcp -d 192.168.201.126@tcp1 -m GET -e remote_dropped -r 2 [ 1372.632755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.26@tcp -d 192.168.201.26@tcp -m GET -e remote_dropped -r 2 [ 1372.673261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.26@tcp -d 192.168.201.26@tcp1 -m GET -e remote_dropped -r 2 [ 1372.708861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.26@tcp1 -d 192.168.201.126@tcp -m GET -e remote_dropped -r 2 [ 1372.760903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.26@tcp1 -d 192.168.201.126@tcp1 -m GET -e remote_dropped -r 2 [ 1372.803433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.26@tcp1 -d 192.168.201.26@tcp -m GET -e remote_dropped -r 2 [ 1372.848389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.26@tcp1 -d 192.168.201.26@tcp1 -m GET -e remote_dropped -r 2 [ 1372.891600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.26@tcp 192.168.201.126@tcp [ 1372.913861] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.126@tcp [ 1373.196393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1373.232687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1373.273491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1373.296801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1373.346921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1373.382526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1373.409450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1373.448251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1373.506902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1373.573637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1373.607755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1373.666474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1373.718673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1373.766733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1378.911588] LNet: 86678:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1378.917618] LNetError: 86678:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1378.931142] LNet: Removed LNI 192.168.201.26@tcp [ 1378.939306] LNet: Removed LNI 192.168.201.26@tcp1 [ 1379.320295] Key type .llcrypt unregistered [ 1379.324149] Key type ._llcrypt unregistered [ 1398.807315] Key type ._llcrypt registered [ 1398.812745] Key type .llcrypt registered [ 1399.951364] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 12:46:49 (1780332409) [ 1400.222921] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1400.240976] alg: No test for adler32 (adler32-zlib) [ 1401.204498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1401.254653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1401.271045] LNet: Added LNI 192.168.201.26@tcp [8/256/0/180] [ 1402.046155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1402.069367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.201.26@tcp [ 1402.083312] LNetError: 87660:0:(api-ni.c:4350:lnet_ni_set_healthv()) manually adding local NI 192.168.201.26@tcp to recovery [ 1402.185150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.201.26@tcp [ 1402.358978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1408.497875] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 12:46:58 (1780332418) [ 1408.720623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1408.742606] LNet: 88250:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1408.747656] LNetError: 88250:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1408.758391] LNet: Removed LNI 192.168.201.26@tcp [ 1408.784597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1408.838785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1408.862516] LNet: Added LNI 192.168.201.26@tcp [8/256/0/180] [ 1409.477475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1409.495759] LNet: Added LNI 192.168.201.26@tcp1 [8/256/0/180] [ 1410.257632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.201.26@tcp [ 1410.271464] LNetError: 88516:0:(api-ni.c:4350:lnet_ni_set_healthv()) manually adding local NI 192.168.201.26@tcp to recovery [ 1410.847480] LNet: 1 local NIs in recovery (showing 1): 192.168.201.26@tcp [ 1410.856267] LNet: There was an unexpected network error while writing to 192.168.201.26: rc = -22 [ 1411.536583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.201.26@tcp [ 1420.236975] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 12:47:09 (1780332429) [ 1422.048443] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg126-server [ 1424.082638] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 12:47:13 (1780332433) [ 1426.195669] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg126-server [ 1427.938563] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 12:47:17 (1780332437) [ 1430.013861] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg126-server [ 1431.866491] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 12:47:21 (1780332441) [ 1431.996854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1432.031293] LNet: 90141:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1432.041625] LNetError: 90141:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1432.050720] LNet: Removed LNI 192.168.201.26@tcp [ 1432.063993] LNet: Removed LNI 192.168.201.26@tcp1 [ 1432.088350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1432.159261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1432.174692] LNet: Added LNI 192.168.201.26@tcp [8/256/0/180] [ 1432.857869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1432.923748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1432.967177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1433.055382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1433.137450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1433.205817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1439.535488] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 12:47:29 (1780332449) [ 1439.956846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1439.992143] LNet: 90872:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1440.002740] LNetError: 90872:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1440.012294] LNetError: 90872:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1440.024498] LNet: Removed LNI 192.168.201.26@tcp [ 1440.083688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1440.173719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1440.203229] LNet: Added LNI 192.168.201.26@tcp [8/256/0/180] [ 1447.823411] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 12:47:37 (1780332457) [ 1448.240307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1448.273501] LNet: 91576:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1448.278798] LNetError: 91576:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1448.291942] LNet: Removed LNI 192.168.201.26@tcp [ 1448.333198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1448.445087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1448.493694] LNet: Added LNI 192.168.201.26@tcp [8/256/0/180] [ 1457.038751] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 12:47:46 (1780332466) [ 1459.319954] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 1461.126674] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 12:47:50 (1780332470) [ 1461.650478] LNet: 92611:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1461.657697] LNetError: 92611:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1461.672162] LNet: Removed LNI 192.168.201.26@tcp [ 1462.178860] Key type .llcrypt unregistered [ 1462.180882] Key type ._llcrypt unregistered [ 1468.219643] Key type ._llcrypt registered [ 1468.220934] Key type .llcrypt registered [ 1468.331858] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1468.343414] alg: No test for adler32 (adler32-zlib) [ 1469.284209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1469.302845] LNet: Added LNI 192.168.201.26@tcp [8/256/0/180] [ 1478.353339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.126@tcp [ 1484.030598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.201.126@tcp [ 1484.533738] LNet: 93029:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1484.541393] LNetError: 93029:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1484.577200] LNet: Removed LNI 192.168.201.26@tcp [ 1485.169530] Key type .llcrypt unregistered [ 1485.171935] Key type ._llcrypt unregistered [ 1498.941607] Key type ._llcrypt registered [ 1498.943263] Key type .llcrypt registered [ 1500.372796] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 1502.487583] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 1504.267152] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 12:48:33 (1780332513) [ 1506.100642] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 [ 1508.372251] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 12:48:37 (1780332517) [ 1510.400539] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg126-server [ 1511.664785] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 12:48:41 (1780332521) [ 1513.301403] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg126-server [ 1515.196859] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 12:48:44 (1780332524) [ 1517.217384] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1517.237363] alg: No test for adler32 (adler32-zlib) [ 1518.382763] Lustre: Lustre: Build Version: 2.17.53_31_g0c7f08d [ 1518.610295] LNet: Added LNI 192.168.201.26@tcp [8/256/0/180] [ 1520.303225] Key type lgssc registered [ 1521.733367] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1543.842160] Key type lgssc unregistered [ 1544.088588] LNet: 96206:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1544.100379] LNetError: 96206:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1544.115693] LNet: Removed LNI 192.168.201.26@tcp [ 1544.771696] Key type .llcrypt unregistered [ 1544.773291] Key type ._llcrypt unregistered [ 1560.196954] Key type ._llcrypt registered [ 1560.204889] Key type .llcrypt registered [ 1561.818527] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 12:49:30 (1780332570) [ 1562.363357] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1562.389371] alg: No test for adler32 (adler32-zlib) [ 1563.389891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1563.477606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1563.540199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1563.573360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1563.621760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1569.893751] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 12:49:39 (1780332579) [ 1570.048775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1570.074170] LNet: 97621:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1570.094915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1570.151152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1570.184827] LNet: Added LNI 192.168.201.26@tcp [8/256/0/180] [ 1571.240397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1571.252102] LNet: 97765:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1571.257928] LNetError: 97765:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1571.273180] LNet: Removed LNI 192.168.201.26@tcp [ 1571.294882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1571.347509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1571.374501] LNet: Added LNI 192.168.201.26@tcp [8/256/0/180] [ 1571.915636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1571.934634] LNetError: 97793:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1571.948408] LNet: Removed LNI 192.168.201.26@tcp [ 1571.979779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1572.080525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1572.114954] LNet: Added LNI 192.168.201.26@tcp [8/256/0/180] [ 1579.224598] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 12:49:48 (1780332588) [ 1579.502264] LNet: 98384:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1579.506556] LNet: 98384:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1579.515547] LNetError: 98384:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1579.531937] LNet: Removed LNI 192.168.201.26@tcp [ 1580.232237] Key type .llcrypt unregistered [ 1580.238638] Key type ._llcrypt unregistered [ 1586.631331] Key type ._llcrypt registered [ 1586.637844] Key type .llcrypt registered [ 1586.950206] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1586.990494] alg: No test for adler32 (adler32-zlib) [ 1587.970414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1588.124157] LNet: Added LNI 192.168.201.26@tcp [8/256/0/180] [ 1588.128506] LNet: *** cfs_fail_loc=e003, val=100*** [ 1588.953890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.26@tcp [ 1588.984416] LNet: *** cfs_fail_loc=e003, val=100*** [ 1588.988740] LNet: Skipped 2 previous similar messages [ 1589.005221] Ping reply received with mapping: [ 1589.006775] found metadata reply NI[192.168.201.26@tcp] [ 1589.010665] 1a c9 a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 1589.013628] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 1589.020425] 5a 5a 5a 5a 5a 5a 5a 5a [ 1589.140432] LNet: 98782:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1589.144840] LNetError: 98782:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1589.161698] LNet: Removed LNI 192.168.201.26@tcp [ 1589.794625] Key type .llcrypt unregistered [ 1589.797405] Key type ._llcrypt unregistered [ 1601.976637] Key type ._llcrypt registered [ 1601.979245] Key type .llcrypt registered [ 1603.480837] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 12:50:12 (1780332612) [ 1604.231278] Key type .llcrypt unregistered [ 1604.233743] Key type ._llcrypt unregistered [ 1611.653459] Key type ._llcrypt registered [ 1611.654566] Key type .llcrypt registered [ 1611.948687] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1611.967786] alg: No test for adler32 (adler32-zlib) [ 1613.025749] LNetError: 99830:0:(api-ni.c:2729:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 1613.343776] Key type .llcrypt unregistered [ 1613.345554] Key type ._llcrypt unregistered [ 1618.660415] Key type ._llcrypt registered [ 1618.664520] Key type .llcrypt registered [ 1620.140363] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 12:50:29 (1780332629) [ 1620.790827] Key type .llcrypt unregistered [ 1620.796861] Key type ._llcrypt unregistered [ 1628.721991] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1628.735973] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1631.548477] Key type ._llcrypt registered [ 1631.550785] Key type .llcrypt registered [ 1631.934330] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1631.953828] alg: No test for adler32 (adler32-zlib) [ 1632.955311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1633.025901] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1643.460536] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 12:50:52 (1780332652) [ 1643.536175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1643.558442] LNet: 101603:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1643.571102] LNetError: 101603:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1643.595146] LNet: Removed LNI 10.1.2.3@tcp [ 1643.628319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1644.321203] Key type .llcrypt unregistered [ 1644.325485] Key type ._llcrypt unregistered [ 1651.312669] Key type ._llcrypt registered [ 1651.317503] Key type .llcrypt registered [ 1651.512219] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1651.540652] alg: No test for adler32 (adler32-zlib) [ 1652.467116] LNet: Added LNI 192.168.201.26@tcp [8/256/0/180] [ 1652.566827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.26@tcp [ 1652.725516] LNet: 101899:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1652.731800] LNetError: 101899:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1652.752927] LNet: Removed LNI 192.168.201.26@tcp [ 1653.359320] Key type .llcrypt unregistered [ 1653.372106] Key type ._llcrypt unregistered [ 1664.515834] Key type ._llcrypt registered [ 1664.517854] Key type .llcrypt registered [ 1665.813394] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 12:51:15 (1780332675) [ 1666.620308] Key type .llcrypt unregistered [ 1666.621862] Key type ._llcrypt unregistered [ 1672.648506] Key type ._llcrypt registered [ 1672.650595] Key type .llcrypt registered [ 1672.855233] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1672.879432] alg: No test for adler32 (adler32-zlib) [ 1679.671366] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 12:51:29 (1780332689) [ 1679.809337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1679.863792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1679.949596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1679.996443] LNet: Added LNI 192.168.201.26@tcp [8/256/0/180] [ 1680.750050] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1680.797766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 1680.827994] LNet: Removed LNI 192.168.201.26@tcp [ 1681.181727] LNet: 103642:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1681.448398] Key type .llcrypt unregistered [ 1681.449886] Key type ._llcrypt unregistered [ 1691.638636] Key type ._llcrypt registered [ 1691.641321] Key type .llcrypt registered [ 1693.028961] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 12:51:42 (1780332702) [ 1693.475593] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1693.491323] alg: No test for adler32 (adler32-zlib) [ 1694.434734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1694.482685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1694.507608] LNet: Added LNI 192.168.201.26@tcp [8/256/0/180] [ 1695.174994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.26@tcp [ 1695.205645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.201.26@tcp [ 1695.218149] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 1695.224317] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 1695.226806] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1695, 0, 0 [ 1695.230479] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 1695.232725] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 1695.235561] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 1695.238347] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1695, 0, 0 [ 1695.241455] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 1695.243552] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1695.246559] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 1695.249281] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1695, 0, 1745 [ 1695.252721] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 1695.254796] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1695.257406] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 1695.260275] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1695, 0, 1745 [ 1695.263717] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 1700.928622] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 12:51:50 (1780332710) [ 1701.674511] LNet: 105181:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1701.683983] LNetError: 105181:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1701.695304] LNet: Removed LNI 192.168.201.26@tcp [ 1702.229467] Key type .llcrypt unregistered [ 1702.232204] Key type ._llcrypt unregistered [ 1708.008949] Key type ._llcrypt registered [ 1708.013316] Key type .llcrypt registered [ 1708.211051] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1708.240989] alg: No test for adler32 (adler32-zlib) [ 1709.190604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1709.230929] LNet: Added LNI 192.168.201.26@tcp [8/256/0/180] [ 1716.093458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.126@tcp [ 1726.055456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1726.087269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1726.106608] LNet: Added LNI 192.168.201.26@tcp1 [8/256/0/180] [ 1726.949128] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1727.052505] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1727.059480] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1727.065881] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1729.464861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 1729.483288] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 1739.452348] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 12:52:28 (1780332748) [ 1739.574277] LNet: 106688:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1739.585928] LNetError: 106688:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1739.593159] LNet: Removed LNI 192.168.201.26@tcp [ 1740.639351] LNet: Removed LNI 192.168.201.26@tcp1 [ 1740.642106] LNetError: 106688:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1740.646346] LNetError: 106688:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1740.666733] LNet: Removed LNI 10.1.2.3@tcp99 [ 1741.513400] Key type .llcrypt unregistered [ 1741.516610] Key type ._llcrypt unregistered [ 1747.719098] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1747.726689] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1750.277478] Key type ._llcrypt registered [ 1750.279720] Key type .llcrypt registered [ 1750.502040] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1750.521082] alg: No test for adler32 (adler32-zlib) [ 1751.519553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1751.562851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1751.586187] LNet: Added LNI 192.168.201.26@tcp [8/256/0/180] [ 1752.392202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1752.441242] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1754.876895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 1754.923304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1754.955496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 1754.973520] LNetError: 107436:0:(peer.c:2175:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 1755.015262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1755.053961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.26@tcp [ 1755.545427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1755.596204] LNet: 107457:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1755.599868] LNetError: 107457:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1756.640056] LNet: Removed LNI 192.168.201.26@tcp [ 1756.644547] LNetError: 107457:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1756.659901] LNet: Removed LNI 10.1.2.3@tcp [ 1757.381592] Key type .llcrypt unregistered [ 1757.384344] Key type ._llcrypt unregistered [ 1769.854923] Key type ._llcrypt registered [ 1769.858570] Key type .llcrypt registered [ 1771.207182] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 12:53:00 (1780332780) [ 1771.544828] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1771.561929] alg: No test for adler32 (adler32-zlib) [ 1772.486981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1772.540896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1772.565436] LNet: Added LNI 192.168.201.26@tcp [8/256/0/180] [ 1773.438481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.26@tcp [ 1773.567528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg126-client@tcp [ 1779.264976] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 12:53:08 (1780332788) [ 1779.353352] LNet: 108983:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1779.364353] LNetError: 108983:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1780.390883] LNet: Removed LNI 192.168.201.26@tcp [ 1781.007694] Key type .llcrypt unregistered [ 1781.017304] Key type ._llcrypt unregistered [ 1787.355762] Key type ._llcrypt registered [ 1787.358691] Key type .llcrypt registered [ 1787.593906] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1787.629872] alg: No test for adler32 (adler32-zlib) [ 1788.599044] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 1788.654033] LNet: Added LNI 192.168.201.26@tcp [8/256/0/180] [ 1788.695849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1797.157688] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 12:53:26 (1780332806) [ 1799.592976] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 1801.748596] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 12:53:30 (1780332810) [ 1803.806156] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 1805.603354] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 12:53:35 (1780332815) [ 1805.902819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1805.939625] LNet: 110412:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1805.967344] LNetError: 110412:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1805.992353] LNet: Removed LNI 192.168.201.26@tcp [ 1806.012540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1806.095691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1806.134922] LNet: Added LNI 192.168.201.26@tcp [8/256/0/180] [ 1807.146567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 1807.191413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 1817.041697] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 12:53:45 (1780332825) [ 1817.191104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1817.256657] LNet: 111116:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1817.279678] LNetError: 111116:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1818.343597] LNet: Removed LNI 192.168.201.26@tcp [ 1818.388475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1818.478913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 1818.545557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1818.823045] LNet: 111159:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1819.349674] Key type .llcrypt unregistered [ 1819.354231] Key type ._llcrypt unregistered [ 1832.894190] Key type ._llcrypt registered [ 1832.897327] Key type .llcrypt registered [ 1834.313994] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 12:54:03 (1780332843) [ 1834.805736] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1834.833273] alg: No test for adler32 (adler32-zlib) [ 1835.831438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1835.905031] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1835.944955] LNet: Added LNI 192.168.201.26@tcp [8/256/0/180] [ 1836.732393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1836.816656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.26@tcp [ 1844.983241] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 12:54:14 (1780332854) [ 1845.230069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1845.267988] LNet: 112677:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1845.282622] LNetError: 112677:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1845.314141] LNet: Removed LNI 192.168.201.26@tcp [ 1845.345145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1845.401321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 1845.428765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 1853.584461] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 12:54:22 (1780332862) [ 1853.656041] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1853.698303] LNet: 113251:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1853.725680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1853.886082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 1854.585819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1854.633521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1854.725213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 1855.421452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1855.453930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1855.556162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1855.598487] LNet: 113293:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1855.621423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1855.719495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1856.199645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1856.263728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1856.422706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 1857.104669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1857.163566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1864.813420] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 12:54:34 (1780332874) [ 1864.966229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1865.021502] LNet: 113892:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1865.084688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1865.179444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1865.200990] LNet: Added LNI 192.168.201.26@tcp [8/256/0/180] [ 1865.950676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1866.002996] LNet: Added LNI 192.168.201.26@tcp2 [8/256/0/180] [ 1866.793585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 1866.874387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.26@tcp [ 1867.074759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1867.095884] LNetError: 114173:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1868.127254] LNet: Removed LNI 192.168.201.26@tcp [ 1868.133183] LNetError: 114173:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1868.154548] LNet: Removed LNI 192.168.201.26@tcp2 [ 1868.195769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1868.293523] LNet: Added LNI 192.168.201.26@tcp [8/256/0/180] [ 1868.309301] LNet: Added LNI 192.168.201.26@tcp2 [8/256/0/180] [ 1868.322570] LNet: 114179:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1869.061723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.26@tcp [ 1869.246497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1869.272378] LNet: 114216:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1869.278137] LNet: 114216:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1870.303351] LNet: Removed LNI 192.168.201.26@tcp [ 1870.308547] LNetError: 114216:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1870.321058] LNetError: 114216:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1870.372079] LNet: Removed LNI 192.168.201.26@tcp2 [ 1870.437992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1870.532864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1870.578191] LNet: Added LNI 192.168.201.26@tcp [8/256/0/180] [ 1871.669216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1871.703563] LNet: Added LNI 192.168.201.26@tcp2 [8/256/0/180] [ 1872.647255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.26@tcp [ 1872.700649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 1872.788182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.26@tcp [ 1873.130178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1874.207503] LNet: Removed LNI 192.168.201.26@tcp [ 1875.304513] LNet: Removed LNI 192.168.201.26@tcp2 [ 1875.376814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1875.553971] LNet: Added LNI 192.168.201.26@tcp [8/256/0/180] [ 1875.564621] LNet: Added LNI 192.168.201.26@tcp2 [8/256/0/180] [ 1875.573611] LNet: 114507:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1876.137277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.26@tcp [ 1876.544160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1876.574510] LNetError: 114544:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1876.582515] LNetError: 114544:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1876.594137] LNet: Removed LNI 192.168.201.26@tcp [ 1877.609973] LNet: Removed LNI 192.168.201.26@tcp2 [ 1877.627332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1877.677485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1877.697287] LNet: Added LNI 192.168.201.26@tcp [8/256/0/180] [ 1878.915925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1878.934571] LNet: Added LNI 192.168.201.26@tcp2 [8/256/0/180] [ 1879.886444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1879.975846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.26@tcp [ 1880.247332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1880.268327] LNet: 114824:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1880.272267] LNet: 114824:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 1881.312231] LNet: Removed LNI 192.168.201.26@tcp [ 1881.338677] LNet: Removed LNI 192.168.201.26@tcp2 [ 1881.377515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1881.519564] LNet: Added LNI 192.168.201.26@tcp [8/256/0/180] [ 1881.538078] LNet: Added LNI 192.168.201.26@tcp2 [8/256/0/180] [ 1881.548110] LNet: 114830:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1882.638946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.26@tcp [ 1883.034584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1884.135577] LNet: Removed LNI 192.168.201.26@tcp [ 1884.149606] LNet: Removed LNI 192.168.201.26@tcp2 [ 1884.192421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1884.300447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1884.339455] LNet: Added LNI 192.168.201.26@tcp [8/256/0/180] [ 1885.539961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1885.565083] LNet: Added LNI 192.168.201.26@tcp2 [8/256/0/180] [ 1886.773412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.26@tcp [ 1886.809504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 1886.894333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.26@tcp [ 1887.289488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1887.339121] LNetError: 115152:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1887.348314] LNetError: 115152:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 1887.355718] LNet: Removed LNI 192.168.201.26@tcp [ 1888.424199] LNet: Removed LNI 192.168.201.26@tcp2 [ 1888.465866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1888.530630] LNet: Added LNI 192.168.201.26@tcp [8/256/0/180] [ 1888.536236] LNet: Added LNI 192.168.201.26@tcp2 [8/256/0/180] [ 1888.541654] LNet: 115159:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1889.286826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.26@tcp [ 1889.517502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1889.534203] LNet: Removed LNI 192.168.201.26@tcp [ 1890.608085] LNet: Removed LNI 192.168.201.26@tcp2 [ 1890.660756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1890.747614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1890.780636] LNet: Added LNI 192.168.201.26@tcp [8/256/0/180] [ 1891.735887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1891.755447] LNet: Added LNI 192.168.201.26@tcp2 [8/256/0/180] [ 1892.802493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 1892.911602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1893.021683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.26@tcp [ 1893.386800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1894.431627] LNet: Removed LNI 192.168.201.26@tcp [ 1894.446991] LNet: Removed LNI 192.168.201.26@tcp2 [ 1894.486854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1894.599273] LNet: Added LNI 192.168.201.26@tcp [8/256/0/180] [ 1894.606939] LNet: Added LNI 192.168.201.26@tcp2 [8/256/0/180] [ 1894.623244] LNet: 115499:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1895.110826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.26@tcp [ 1895.551921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1896.607261] LNet: Removed LNI 192.168.201.26@tcp [ 1896.628521] LNet: Removed LNI 192.168.201.26@tcp2 [ 1896.681152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1896.774970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1896.800071] LNet: Added LNI 192.168.201.26@tcp [8/256/0/180] [ 1897.670714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1897.690969] LNet: Added LNI 192.168.201.26@tcp2 [8/256/0/180] [ 1898.513392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.26@tcp [ 1898.557882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 1898.600482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 1898.648474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.26@tcp [ 1898.974384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1898.995177] LNet: 115853:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1898.999667] LNet: 115853:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 5 previous similar messages [ 1900.063155] LNet: Removed LNI 192.168.201.26@tcp [ 1901.093258] LNet: Removed LNI 192.168.201.26@tcp2 [ 1901.139805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1901.240378] LNet: Added LNI 192.168.201.26@tcp [8/256/0/180] [ 1901.252900] LNet: Added LNI 192.168.201.26@tcp2 [8/256/0/180] [ 1901.963350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.26@tcp [ 1908.889327] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 1910.631955] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 12:55:20 (1780332920) [ 1910.958550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1910.977580] LNetError: 116619:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1910.982818] LNetError: 116619:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 9 previous similar messages [ 1910.991636] LNet: Removed LNI 192.168.201.26@tcp [ 1912.037516] LNet: Removed LNI 192.168.201.26@tcp2 [ 1912.096525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1912.490407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1912.508151] LNet: Added LNI 192.168.201.26@tcp [8/256/0/180] [ 1913.249776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1913.267617] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1914.034151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1914.045820] LNet: Removed LNI 192.168.201.26@tcp [ 1914.058307] LNet: Removed LNI 10.1.2.3@tcp [ 1914.069579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1914.134720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 1914.166148] LNet: Added LNI 192.168.201.26@tcp [8/256/0/180] [ 1914.171805] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1914.701905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 1914.724039] LNet: Removed LNI 192.168.201.26@tcp [ 1914.732386] LNet: Removed LNI 10.1.2.3@tcp [ 1921.967522] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 12:55:31 (1780332931) [ 1922.266161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1922.331342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1922.441929] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1922.553776] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1922.566769] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1922.581317] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1925.268505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1925.312764] LNet: Added LNI 192.168.201.26@tcp [8/256/0/180] [ 1926.446763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1926.482827] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1929.908511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.26@tcp [ 1929.992898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 1930.097612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.201.26@tcp -m PUT -e local_timeout -r 1 [ 1930.222447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 1930.311526] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1930.314344] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1930.522130] LNetError: 117559:0:(lib-move.c:4710:lnet_parse()) 192.168.201.26@tcp, src 192.168.201.26@tcp: Dropping PUT (error -108 looking up sender) [ 1931.558289] LNet: 118018:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1931.565130] LNet: 118018:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 1932.639179] LNet: Removed LNI 192.168.201.26@tcp [ 1932.651511] LNet: Removed LNI 10.1.2.3@tcp [ 1933.242923] Key type .llcrypt unregistered [ 1933.249656] Key type ._llcrypt unregistered [ 1948.935287] Key type ._llcrypt registered [ 1948.938130] Key type .llcrypt registered [ 1950.877445] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 12:55:59 (1780332959) [ 1951.422737] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1951.448246] alg: No test for adler32 (adler32-zlib) [ 1952.537438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1953.031286] LNet: 118872:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1953.607358] Key type .llcrypt unregistered [ 1953.608651] Key type ._llcrypt unregistered [ 1961.172358] Key type ._llcrypt registered [ 1961.183729] Key type .llcrypt registered [ 1961.548656] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1961.576948] alg: No test for adler32 (adler32-zlib) [ 1962.490100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1962.511412] LNet: Added LNI 192.168.201.26@tcp [8/256/0/180] [ 1971.659417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.126@tcp [ 1971.956872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1972.683705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.201.126@tcp [ 1972.768471] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.126@tcp [ 1972.990933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1973.018749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1973.050275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1973.080879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1973.108662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1973.140571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1973.211915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1973.258896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1973.292903] LNet: 119323:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1973.320612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1973.410585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1973.487685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1973.534241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1973.585227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1973.663233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1974.696598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 1975.034354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1975.086423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1975.126500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1975.196768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1975.249955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1975.286990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1975.335913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1975.389234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1975.430877] LNet: 119440:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1975.463822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1975.522178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1975.581310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1975.642731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1975.713297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1975.749235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1976.639625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.201.126@tcp [ 1976.863993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1976.900735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1976.933324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1976.977535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1977.023254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1977.065428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1977.117766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1977.192303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1977.218375] LNet: 119558:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1977.233606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1977.276657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1977.332698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1977.402450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1977.463581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1977.509361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1978.475549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 1978.827733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1978.868158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1978.917463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1978.962272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1978.997959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1979.029703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1979.070801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1979.141526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1979.230433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1979.288527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1979.357938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1979.420779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1979.469976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1979.503809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1979.861518] LNet: 119721:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1979.864919] LNetError: 119721:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1979.888330] LNet: Removed LNI 192.168.201.26@tcp [ 1980.391406] Key type .llcrypt unregistered [ 1980.398669] Key type ._llcrypt unregistered [ 2001.070648] Key type ._llcrypt registered [ 2001.072683] Key type .llcrypt registered [ 2002.615148] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 12:56:51 (1780333011) [ 2003.138248] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2003.154173] alg: No test for adler32 (adler32-zlib) [ 2004.176518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2004.716159] LNet: 120610:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2005.668512] Key type .llcrypt unregistered [ 2005.670709] Key type ._llcrypt unregistered [ 2013.334644] Key type ._llcrypt registered [ 2013.336125] Key type .llcrypt registered [ 2013.595899] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2013.615607] alg: No test for adler32 (adler32-zlib) [ 2014.675790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2014.744948] LNet: Added LNI 192.168.201.26@tcp [8/256/0/180] [ 2023.008478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.126@tcp [ 2034.605133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2034.650664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2034.690317] LNet: Added LNI 192.168.201.26@tcp1 [8/256/0/180] [ 2035.710525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 2036.512083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.201.126@tcp [ 2036.794037] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2036.819837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2036.860433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2036.910034] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2036.951155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2036.978784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2037.021816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2037.083184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2037.135898] LNet: 121263:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2037.164363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2037.244408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2037.281399] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.126@tcp [ 2037.311905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2037.358835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2037.440896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2037.474372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2038.214830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2038.532723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2038.575679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2038.629778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2038.657764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2038.683356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2038.709706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2038.741958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2038.781547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2038.801234] LNet: 121380:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2038.822343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2038.863760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2038.901342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2038.967177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2039.012475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2039.058523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2039.730431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.201.126@tcp [ 2039.975717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2040.010389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2040.043480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2040.077466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2040.150446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2040.238233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2040.272107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2040.318506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2040.347664] LNet: 121497:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2040.374775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2040.422493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2040.464647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2040.511485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2040.549597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2040.593588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2041.313741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2041.606809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2041.637245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2041.663127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2041.701715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2041.749362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2041.793188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2041.846381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2041.888775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2041.945877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2041.994584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2042.059746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2042.101972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2042.154067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2042.214449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2047.302844] LNet: 121695:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2047.308981] LNetError: 121695:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2047.316574] LNet: Removed LNI 192.168.201.26@tcp [ 2047.325741] LNet: Removed LNI 192.168.201.26@tcp1 [ 2047.787958] Key type .llcrypt unregistered [ 2047.794793] Key type ._llcrypt unregistered [ 2069.014462] Key type ._llcrypt registered [ 2069.017393] Key type .llcrypt registered [ 2070.730097] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 12:57:59 (1780333079) [ 2080.304807] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 12:58:09 (1780333089) [ 2088.864762] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 12:58:18 (1780333098) [ 2098.297533] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 12:58:27 (1780333107) [ 2107.252438] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 12:58:36 (1780333116) [ 2116.949754] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 12:58:45 (1780333125) [ 2126.790398] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 12:58:55 (1780333135) [ 2135.009817] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 12:59:04 (1780333144) [ 2142.864452] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 12:59:12 (1780333152) [ 2151.246179] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 12:59:20 (1780333160) [ 2159.325806] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 12:59:28 (1780333168) [ 2168.362701] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 12:59:37 (1780333177) [ 2176.456225] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 12:59:45 (1780333185) [ 2183.980221] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 12:59:53 (1780333193) [ 2192.199741] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 13:00:01 (1780333201) [ 2199.949688] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 13:00:09 (1780333209) [ 2207.940216] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 13:00:16 (1780333216) [ 2217.016425] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 13:00:26 (1780333226) [ 2217.488321] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2217.505274] alg: No test for adler32 (adler32-zlib) [ 2218.528311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2218.714648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2218.755412] LNet: Added LNI 192.168.201.26@tcp [8/256/0/180] [ 2219.949773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2219.984920] LNet: 132412:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2219.994931] LNetError: 132412:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2220.013454] LNet: Removed LNI 192.168.201.26@tcp [ 2220.039697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2220.115475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2220.155988] LNet: Added LNI 192.168.201.26@tcp [8/256/0/180] [ 2228.600307] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 13:00:37 (1780333237) [ 2236.133274] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 13:00:45 (1780333245) [ 2245.237463] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 13:00:54 (1780333254) [ 2254.348854] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 13:01:03 (1780333263) [ 2262.852343] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 13:01:12 (1780333272) [ 2271.987408] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 13:01:21 (1780333281) [ 2280.161467] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 13:01:29 (1780333289) [ 2288.307651] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 13:01:37 (1780333297) [ 2297.635754] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 13:01:46 (1780333306) [ 2307.357885] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 13:01:56 (1780333316) [ 2315.127149] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 13:02:04 (1780333324) [ 2323.329701] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 13:02:12 (1780333332) [ 2331.085545] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 2232 sec ========== 13:02:20 (1780333340) [ 2331.519940] LNet: 139894:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2331.526517] LNetError: 139894:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2331.538863] LNet: Removed LNI 192.168.201.26@tcp [ 2332.221641] Key type .llcrypt unregistered [ 2332.228391] Key type ._llcrypt unregistered [ 2348.552153] Key type ._llcrypt registered [ 2348.554257] Key type .llcrypt registered [ 2348.957951] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2348.969560] alg: No test for adler32 (adler32-zlib) [ 2350.261573] Lustre: Lustre: Build Version: 2.17.53_31_g0c7f08d [ 2350.669616] LNet: Added LNI 192.168.201.26@tcp [8/256/0/180] [ 2352.447261] Key type lgssc registered [ 2354.276655] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2423.393236] Lustre: Mounted lustre-client [ 2429.355585] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2438.930827] LustreError: 142344:0:(lov_obd.c:792:lov_cleanup()) lustre-clilov-ffff9056e259f800: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 2439.004061] Lustre: Unmounted lustre-client [ 2479.361351] Key type lgssc unregistered [ 2479.628715] LNet: 142827:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2479.637630] LNetError: 142827:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2479.663307] LNet: Removed LNI 192.168.201.26@tcp [ 2480.332181] Key type .llcrypt unregistered [ 2480.334272] Key type ._llcrypt unregistered