[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 457277021 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2421 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE22BD 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 00227D (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2331 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE23C1 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE23F9 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe22bd-0xbffe2330] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22bc] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2331-0xbffe23c0] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe23c1-0xbffe23f8] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe23f9-0xbffe2420] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001017] APIC: Switch to symmetric I/O mode setup [ 0.002394] x2apic enabled [ 0.003011] Switched APIC routing to physical x2apic. [ 0.004020] kvm-guest: setup PV IPIs [ 0.007000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007039] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008016] pid_max: default: 32768 minimum: 301 [ 0.010127] LSM: Security Framework initializing [ 0.011070] Yama: becoming mindful. [ 0.012054] SELinux: Initializing. [ 0.013000] *** VALIDATE selinux *** [ 0.020610] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.025444] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.026190] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027144] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028169] *** VALIDATE tmpfs *** [ 0.030413] *** VALIDATE proc *** [ 0.031275] *** VALIDATE cgroup *** [ 0.032016] *** VALIDATE cgroup2 *** [ 0.033316] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.034170] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.035015] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.036045] Spectre V2 : User space: Vulnerable [ 0.037030] Speculative Store Bypass: Vulnerable [ 0.040588] debug: unmapping init [mem 0xffffffff85e59000-0xffffffff85e60fff] [ 0.042375] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.043886] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.044041] ... version: 2 [ 0.045023] ... bit width: 48 [ 0.046022] ... generic registers: 4 [ 0.047020] ... value mask: 0000ffffffffffff [ 0.048024] ... max period: 00007fffffffffff [ 0.049031] ... fixed-purpose events: 3 [ 0.050021] ... event mask: 000000070000000f [ 0.052313] rcu: Hierarchical SRCU implementation. [ 0.054823] smp: Bringing up secondary CPUs ... [ 0.055686] x86: Booting SMP configuration: [ 0.056042] .... node #0, CPUs: #1 #2 #3 [ 0.066216] smp: Brought up 1 node, 4 CPUs [ 0.068015] smpboot: Max logical packages: 1 [ 0.069019] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.092342] node 0 deferred pages initialised in 21ms [ 0.096482] devtmpfs: initialized [ 0.097328] x86/mm: Memory block size: 128MB [ 0.100149] gcov: version magic: 0x41383552 [ 0.102352] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.103097] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.104303] pinctrl core: initialized pinctrl subsystem [ 0.105255] [ 0.105792] ************************************************************* [ 0.106024] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.107019] ** ** [ 0.108020] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.109018] ** ** [ 0.110022] ** This means that this kernel is built to expose internal ** [ 0.111020] ** IOMMU data structures, which may compromise security on ** [ 0.112018] ** your system. ** [ 0.113028] ** ** [ 0.114019] ** If you see this message and you are not debugging the ** [ 0.115018] ** kernel, report this immediately to your vendor! ** [ 0.116022] ** ** [ 0.117019] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.118016] ************************************************************* [ 0.120019] NET: Registered protocol family 16 [ 0.121556] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.122088] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.123093] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.124736] cpuidle: using governor menu [ 0.126338] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.127578] PCI: Using configuration type 1 for base access [ 0.129099] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.136186] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.138033] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.140231] cryptd: max_cpu_qlen set to 1000 [ 0.142215] ACPI: Added _OSI(Module Device) [ 0.143064] ACPI: Added _OSI(Processor Device) [ 0.144007] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.145010] ACPI: Added _OSI(Processor Aggregator Device) [ 0.149000] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.154164] ACPI: Interpreter enabled [ 0.155067] ACPI: PM: (supports S0 S3 S4 S5) [ 0.156015] ACPI: Using IOAPIC for interrupt routing [ 0.157111] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.158325] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.167895] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.168055] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.169025] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.170104] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.172511] acpiphp: Slot [2] registered [ 0.173194] acpiphp: Slot [5] registered [ 0.174195] acpiphp: Slot [6] registered [ 0.175196] acpiphp: Slot [3] registered [ 0.176115] acpiphp: Slot [4] registered [ 0.177120] acpiphp: Slot [7] registered [ 0.178112] acpiphp: Slot [8] registered [ 0.179108] acpiphp: Slot [9] registered [ 0.180181] acpiphp: Slot [10] registered [ 0.181151] acpiphp: Slot [11] registered [ 0.182165] acpiphp: Slot [12] registered [ 0.183128] acpiphp: Slot [13] registered [ 0.184134] acpiphp: Slot [14] registered [ 0.185128] acpiphp: Slot [15] registered [ 0.186112] acpiphp: Slot [16] registered [ 0.187144] acpiphp: Slot [17] registered [ 0.188161] acpiphp: Slot [18] registered [ 0.189070] acpiphp: Slot [19] registered [ 0.189952] acpiphp: Slot [20] registered [ 0.190000] acpiphp: Slot [21] registered [ 0.191066] acpiphp: Slot [22] registered [ 0.192193] acpiphp: Slot [23] registered [ 0.193098] acpiphp: Slot [24] registered [ 0.195104] acpiphp: Slot [25] registered [ 0.196233] acpiphp: Slot [26] registered [ 0.197097] acpiphp: Slot [27] registered [ 0.199090] acpiphp: Slot [28] registered [ 0.200099] acpiphp: Slot [29] registered [ 0.201080] acpiphp: Slot [30] registered [ 0.202207] acpiphp: Slot [31] registered [ 0.204091] PCI host bridge to bus 0000:00 [ 0.205017] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.206048] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.208085] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.211028] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.213028] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.216132] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.218397] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.222000] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.225537] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.232592] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.237038] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.239029] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.242020] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.245023] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.246767] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.249911] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.252049] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.255980] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.261016] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.270018] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.275016] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.280440] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.289133] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.295026] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.307025] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.315411] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.321019] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.331019] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.346030] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.357853] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.360756] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.363525] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.365443] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.368180] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.371242] iommu: Default domain type: Passthrough [ 0.372364] SCSI subsystem initialized [ 0.373175] ACPI: bus type USB registered [ 0.374087] usbcore: registered new interface driver usbfs [ 0.377143] usbcore: registered new interface driver hub [ 0.379147] usbcore: registered new device driver usb [ 0.380266] pps_core: LinuxPPS API ver. 1 registered [ 0.382012] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.384073] PTP clock support registered [ 0.386185] EDAC MC: Ver: 3.0.0 [ 0.388011] PCI: Using ACPI for IRQ routing [ 0.390070] NetLabel: Initializing [ 0.390958] NetLabel: domain hash size = 128 [ 0.391080] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.393092] NetLabel: unlabeled traffic allowed by default [ 0.395415] vgaarb: loaded [ 0.396631] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.398013] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.404225] clocksource: Switched to clocksource kvm-clock [ 0.555565] VFS: Disk quotas dquot_6.6.0 [ 0.556942] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.558982] *** VALIDATE ramfs *** [ 0.559803] *** VALIDATE hugetlbfs *** [ 0.561160] pnp: PnP ACPI init [ 0.563260] pnp: PnP ACPI: found 6 devices [ 0.584815] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.587321] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.588928] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.590653] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.592997] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.595088] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.597856] NET: Registered protocol family 2 [ 0.599936] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.604182] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.607186] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.611643] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.614161] TCP: Hash tables configured (established 65536 bind 65536) [ 0.616363] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.618729] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.621298] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.623968] NET: Registered protocol family 1 [ 0.626819] RPC: Registered named UNIX socket transport module. [ 0.629132] RPC: Registered udp transport module. [ 0.632959] RPC: Registered tcp transport module. [ 0.634848] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.637212] NET: Registered protocol family 44 [ 0.639170] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.641304] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.643631] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.645927] PCI: CLS 0 bytes, default 64 [ 0.647745] Unpacking initramfs... [ 2.815515] debug: unmapping init [mem 0xffff97757cc64000-0xffff97757ffcffff] [ 2.831532] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.839146] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.850802] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.888116] hrtimer: interrupt took 5096435 ns [ 5.159532] Initialise system trusted keyrings [ 5.161239] Key type blacklist registered [ 5.163099] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 5.172698] zbud: loaded [ 5.175600] *** VALIDATE nfs *** [ 5.176782] *** VALIDATE nfs4 *** [ 5.178284] pstore: using deflate compression [ 5.182168] Platform Keyring initialized [ 5.305917] NET: Registered protocol family 38 [ 5.310897] Key type asymmetric registered [ 5.312288] Asymmetric key parser 'x509' registered [ 5.314834] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 5.317806] io scheduler mq-deadline registered [ 5.322662] io scheduler kyber registered [ 5.324302] io scheduler bfq registered [ 5.325720] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 5.328659] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 5.331723] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 5.335404] ACPI: Power Button [PWRF] [ 5.341603] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 5.348410] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 5.359982] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 5.394366] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 5.431424] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 5.444151] Non-volatile memory driver v1.3 [ 5.445920] Linux agpgart interface v0.103 [ 5.493965] virtio_blk virtio1: [vda] 134152 512-byte logical blocks (68.7 MB/65.5 MiB) [ 5.496959] vda: detected capacity change from 0 to 68685824 [ 5.514279] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 5.517719] vdb: detected capacity change from 0 to 1073741824 [ 5.525724] libphy: Fixed MDIO Bus: probed [ 5.533400] usbcore: registered new interface driver usbserial_generic [ 5.536984] usbserial: USB Serial support registered for generic [ 5.540598] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 5.547379] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 5.548903] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 5.551878] mousedev: PS/2 mouse device common for all mice [ 5.556226] rtc_cmos 00:05: RTC can wake from S4 [ 5.560894] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 5.566170] rtc_cmos 00:05: registered as rtc0 [ 5.566476] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 5.566534] intel_pstate: CPU model not supported [ 5.571743] hid: raw HID events driver (C) Jiri Kosina [ 5.580715] usbcore: registered new interface driver usbhid [ 5.582626] usbhid: USB HID core driver [ 5.584991] drop_monitor: Initializing network drop monitor service [ 5.588368] Initializing XFRM netlink socket [ 5.590348] NET: Registered protocol family 10 [ 5.592465] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 5.596410] Segment Routing with IPv6 [ 5.602050] NET: Registered protocol family 17 [ 5.604446] mpls_gso: MPLS GSO support [ 5.611438] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 5.617560] RAS: Correctable Errors collector initialized. [ 5.620195] AVX version of gcm_enc/dec engaged. [ 5.622303] AES CTR mode by8 optimization enabled [ 5.711523] sched_clock: Marking stable (5711487068, 0)->(6680303562, -968816494) [ 5.715694] registered taskstats version 1 [ 5.718138] Loading compiled-in X.509 certificates [ 5.721428] zswap: loaded using pool lzo/zbud [ 5.761920] Key type big_key registered [ 5.780736] Key type encrypted registered [ 5.784945] ima: No TPM chip found, activating TPM-bypass! [ 5.787320] ima: Allocated hash algorithm: sha1 [ 5.788805] ima: No architecture policies found [ 5.791383] evm: Initialising EVM extended attributes: [ 5.793845] evm: security.selinux [ 5.795366] evm: security.ima [ 5.796697] evm: security.capability [ 5.798648] evm: HMAC attrs: 0x1 [ 5.802532] rtc_cmos 00:05: setting system clock to 2026-05-09 00:24:48 UTC (1778286288) [ 5.815641] debug: unmapping init [mem 0xffffffff86e03000-0xffffffff86ffffff] [ 5.820595] debug: unmapping init [mem 0xffffffff85b82000-0xffffffff85e58fff] [ 5.833293] Write protecting the kernel read-only data: 28672k [ 5.861178] debug: unmapping init [mem 0xffffffff84203000-0xffffffff843fffff] [ 5.875811] debug: unmapping init [mem 0xffffffff84b14000-0xffffffff84bfffff] [ 5.977488] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 5.990683] systemd[1]: Detected virtualization kvm. [ 5.993531] systemd[1]: Detected architecture x86-64. [ 5.995731] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 6.047845] systemd[1]: No hostname configured. [ 6.050926] systemd[1]: Set hostname to . [ 6.053536] random: systemd: uninitialized urandom read (16 bytes read) [ 6.056834] systemd[1]: Initializing machine ID from random generator. [ 6.225081] random: ln: uninitialized urandom read (6 bytes read) [ 6.506937] random: systemd: uninitialized urandom read (16 bytes read) [ 6.522684] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 6.535605] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 6.581750] systemd[1]: Starting Create Volatile Files and Directories... Starting Create Volatile Files and Directories... [ OK ] Reached target Initrd Root Device. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Swap. Starting Setup Virtual Console... [ OK ] Listening on udev Control Socket. [ OK ] Started Memstrack Anylazing Service. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Sockets. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Slices. [ OK ] Reached target Paths. Starting Apply Kernel Variables... [ OK ] Reached target Timers. Starting Journal Service... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 8.418402] device-mapper: uevent: version 1.0.3 [ 8.424601] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 9.923051] virtio_net virtio0 ens2: renamed from eth0 [ 10.400104] scsi host0: ata_piix [ 10.882222] scsi host1: ata_piix [ 10.902453] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 10.904626] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 15.327468] random: crng init done [ 15.328834] random: 7 urandom warning(s) missed due to ratelimiting [ 18.936730] dracut-initqueue[587]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 21.252096] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 25.219974] printk: systemd: 26 output lines suppressed due to ratelimiting [ 26.464511] SELinux: Disabled at runtime. [ 26.657413] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 26.679198] systemd[1]: Detected virtualization kvm. [ 26.681597] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 28.876319] systemd[1]: initrd-switch-root.service: Succeeded. [ 28.879733] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 28.902410] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 28.913636] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 28.921023] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 28.947700] systemd[1]: Starting Journal Service... Starting Journal Service... [ 28.960222] systemd[1]: proc-sys-fs-binfmt_misc.automount: Refusing to start, unit to trigger not loaded. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Created slice system-getty.slice. Mounting Huge Pages File System... [ OK ] Started Forward Password Requests to Wall Directory Watch. Mounting Kernel Debug File System... [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Created slice system-sshd\x2dkeygen.slice. Starting Remount Root and Kernel File Systems... [ OK ] Listening on udev Kernel Socket. Starting Apply Kernel Variables... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... Activating swap /dev/disk/by-label/SWAP... [ OK ] Reached target Paths. [ OK ] Stopped target Initrd File Systems. Mounting POSIX Message Queue File System... Starting Create list of required st…ce nodes for the current kernel... [ 29.640932] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... Starting Configure read-only root support... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ 30.797565] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 32.046033] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 32.054913] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 32.843604] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 32.932261] EDAC sbridge: Ver: 1.1.2 [ 36.010204] Key type dns_resolver registered [* ] A start job is running for Configur…-only root support (8s / no limit)[ 37.074884] NFS: Registering the id_resolver key type [ 37.078230] Key type id_resolver registered [ 37.079998] Key type id_legacy registered [** ] A start job is running for Configur…-only root support (8s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Mark the need to relabel after reboot... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Login Service... [ OK ] Started irqbalance daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. Starting Restore /run/initramfs on shutdown... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Started Login Service. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... [ OK ] Started GSSAPI Proxy Daemon. Starting Hostname Service... [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg116-client login: [ 107.396632] libcfs: loading out-of-tree module taints kernel. [ 107.682377] Key type ._llcrypt registered [ 107.718649] Key type .llcrypt registered [ 108.436343] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 108.467518] alg: No test for adler32 (adler32-zlib) [ 110.031023] Lustre: Lustre: Build Version: 2.17.52_6_gcfb9284 [ 110.959861] LNet: Added LNI 192.168.201.16@tcp [8/256/0/180] [ 112.727292] Key type lgssc registered [ 114.460381] Lustre: Echo OBD driver; http://www.lustre.org/ [ 227.986136] Lustre: Mounted lustre-client [ 233.237863] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 252.871847] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing check_logdir /tmp/testlogs/ [ 253.932261] Lustre: lustre-OST0000-osc-ffff9775c87ee000: disconnect after 23s idle [ 257.067987] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing yml_node [ 260.920541] Lustre: DEBUG MARKER: Client: 2.17.52.6 [ 263.502739] Lustre: DEBUG MARKER: MDS: 2.17.52.6 [ 266.116128] Lustre: DEBUG MARKER: OSS: 2.17.52.6 [ 268.100254] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Fri May 8 20:29:09 EDT 2026 [ 286.628453] LustreError: 7902:0:(lov_obd.c:786:lov_cleanup()) lustre-clilov-ffff9775c87ee000: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 286.663033] Lustre: Unmounted lustre-client [ 312.738050] Key type lgssc unregistered [ 313.096822] LNet: 8349:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 313.109206] LNetError: 8349:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 313.128483] LNet: Removed LNI 192.168.201.16@tcp [ 313.989520] Key type .llcrypt unregistered [ 313.999062] Key type ._llcrypt unregistered [ 338.160430] Key type ._llcrypt registered [ 338.165390] Key type .llcrypt registered [ 338.372347] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 338.399037] alg: No test for adler32 (adler32-zlib) [ 339.408827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 339.480483] LNet: Added LNI 192.168.201.16@tcp [8/256/0/180] [ 339.499841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 344.024611] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 344.314395] LNet: 9231:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 344.319364] LNetError: 9231:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 344.335722] LNet: Removed LNI 192.168.201.16@tcp [ 344.974813] Key type .llcrypt unregistered [ 344.976547] Key type ._llcrypt unregistered [ 351.733290] Key type ._llcrypt registered [ 351.738678] Key type .llcrypt registered [ 353.264344] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 20:30:34 (1778286634) [ 353.664864] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 353.682550] alg: No test for adler32 (adler32-zlib) [ 354.652718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 354.712717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 354.736592] LNet: 9643:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 354.763628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 354.812622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 354.825598] LNet: 9650:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 361.503258] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 20:30:42 (1778286642) [ 361.826879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 361.856690] LNet: 10228:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 361.892168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 362.000900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 369.670282] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 20:30:50 (1778286650) [ 370.061736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 370.101218] LNet: 10815:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 370.140705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 370.239947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 377.420500] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 20:30:58 (1778286658) [ 377.619930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 377.678696] LNet: 11399:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 377.703571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 377.775848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 386.058546] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 20:31:07 (1778286667) [ 386.316724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 386.403932] LNet: 11985:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 386.427782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 386.526312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 386.956372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 394.626543] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 20:31:15 (1778286675) [ 394.925889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 394.958858] LNet: 12576:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 394.983308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 395.097385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 395.871677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 396.531529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 404.708121] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 20:31:25 (1778286685) [ 404.923758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 404.972652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 405.082304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 413.985278] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 20:31:35 (1778286695) [ 414.679428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 414.717031] LNet: 13758:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 414.724489] LNet: 13758:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 414.771627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 414.878473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 414.947977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 415.627563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 415.709295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 416.419244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 416.451498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 417.144043] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 417.193173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 417.832418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 417.902680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 418.658811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 418.726751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 419.278992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 419.360905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 420.057056] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 420.101661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 420.753078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 420.793159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 421.277132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 421.316646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 421.981744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 422.022066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 422.684653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 422.734752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 423.438311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 423.503972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 431.155268] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 20:31:52 (1778286712) [ 431.296137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 431.329645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 431.440803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 431.504849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 439.725877] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 20:32:01 (1778286721) [ 439.863764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 439.932070] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 440.011523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 440.044458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 447.956475] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 20:32:08 (1778286728) [ 448.033537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 448.056231] LNet: 15609:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 448.060622] LNet: 15609:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 448.087825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 448.187891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 448.271705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 455.738554] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 20:32:17 (1778286737) [ 456.300255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 456.397432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 456.526362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 456.585377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 465.262614] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 20:32:26 (1778286746) [ 465.747216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 465.812643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 465.932169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 465.985614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 474.777737] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 20:32:35 (1778286755) [ 474.898250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 474.961479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 475.057904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 475.102627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 481.504189] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 20:32:42 (1778286762) [ 481.712375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 481.793961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 481.952081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 482.470391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 489.677181] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 20:32:50 (1778286770) [ 489.855261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 489.971894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 490.253447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 490.590796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 497.635697] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 20:32:59 (1778286779) [ 497.887629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 497.910303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 498.007979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 498.204672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 504.505096] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 20:33:05 (1778286785) [ 504.628134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 504.704458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 505.328113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 505.600296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 512.553520] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 20:33:13 (1778286793) [ 512.703268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 512.737867] LNet: 20439:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 512.748221] LNet: 20439:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 7 previous similar messages [ 512.798674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 512.915574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 512.966982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 519.552634] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 20:33:20 (1778286800) [ 519.665142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 519.711233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 519.802137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 519.861302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 525.805184] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 20:33:27 (1778286807) [ 525.937945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 525.975470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 526.025290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 532.023789] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 20:33:33 (1778286813) [ 532.246325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 532.290623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 532.377545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 538.212557] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 20:33:39 (1778286819) [ 538.353471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 538.434615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 538.531721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 538.979451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 545.032691] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 20:33:46 (1778286826) [ 545.208903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 545.227674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 545.292679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 545.351775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 551.939080] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 20:33:52 (1778286832) [ 552.001052] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 552.036952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 552.139065] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 552.166525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 558.365585] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 20:33:59 (1778286839) [ 558.661837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 558.711832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 558.775813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 558.803536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 564.818423] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 20:34:06 (1778286846) [ 565.057447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 565.093461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 565.154460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 565.192865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 571.128212] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 20:34:12 (1778286852) [ 571.235179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 571.302737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 571.367211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 571.411340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 571.426947] LNetError: 25760:0:(peer.c:2174:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 571.462631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 571.488741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 577.171316] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 20:34:18 (1778286858) [ 577.286759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 577.379497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 577.509395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.6vvwgCrBE4 [ 577.605339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.6vvwgCrBE4 [ 577.714225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.6vvwgCrBE4 [ 583.323431] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 20:34:24 (1778286864) [ 583.432527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 583.467251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 583.510940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 583.540470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 583.595542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 588.792089] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 20:34:30 (1778286870) [ 589.026098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 589.069699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 589.124669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 591.653706] LNet: Message forwarding will be enabled in 2s [ 594.719139] LNet: Message forwarding enabled [ 595.146581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 595.173353] LNet: Message forwarding will stop in 2s to 6s [ 599.683894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 600.187891] LNet: Message forwarding disabled [ 602.076632] LNet: Message forwarding will be enabled in 2s [ 604.319205] LNet: Message forwarding enabled [ 608.234450] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 20:34:49 (1778286889) [ 608.502879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 609.694718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 609.755666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 615.676684] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 20:34:57 (1778286897) [ 615.914668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 615.963749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 616.008486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 616.034839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 616.061153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 616.086862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 616.110858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 616.141564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 616.178601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 616.208910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 616.235953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 616.262255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 616.295545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 616.323534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 616.354416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 616.383893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 616.419419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 616.463745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 616.515870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 616.551693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 616.585750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 616.602559] LNetError: 28830:0:(router.c:624:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 622.483815] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 20:35:03 (1778286903) [ 622.566150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 622.632814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 622.713802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 628.613869] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 20:35:10 (1778286910) [ 630.231439] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg116-server [ 632.984663] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 20:35:13 (1778286913) [ 633.144461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 633.227965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 633.392340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 633.458359] LNet: Added LNI 192.168.201.16@tcp [8/256/0/180] [ 635.702330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.201.[17-21]@tcp [ 635.730367] LNet: 30546:0:(router.c:728:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 635.771021] LNetError: 30334:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.201.17@tcp is being used as a gateway but routing feature is not turned on [ 635.790049] LNetError: 30334:0:(lib-lnet.h:1336:lnet_set_route_aliveness()) route to tcp1 through 192.168.201.17@tcp has gone from up to down [ 636.785062] LNetError: 30334:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.201.19@tcp is being used as a gateway but routing feature is not turned on [ 636.798994] LNetError: 30334:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 5 previous similar messages [ 637.804822] LNetError: 30334:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.201.17@tcp is being used as a gateway but routing feature is not turned on [ 637.830167] LNetError: 30334:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 2 previous similar messages [ 638.817197] LNetError: 30341:0:(lib-lnet.h:1336:lnet_set_route_aliveness()) route to tcp1 through 192.168.201.18@tcp has gone from up to down [ 638.829362] LNetError: 30341:0:(lib-lnet.h:1336:lnet_set_route_aliveness()) Skipped 2 previous similar messages [ 639.846237] LNetError: 30334:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.201.19@tcp is being used as a gateway but routing feature is not turned on [ 639.864839] LNetError: 30334:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 5 previous similar messages [ 642.436602] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 20:35:23 (1778286923) [ 643.945942] LNetError: 30334:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.201.19@tcp is being used as a gateway but routing feature is not turned on [ 643.965378] LNetError: 30334:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 11 previous similar messages [ 644.080892] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg116-server [ 645.955769] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 20:35:27 (1778286927) [ 646.157124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 646.190623] LNet: 31474:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 646.199976] LNet: 31474:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 16 previous similar messages [ 646.233874] LNetError: 31474:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 646.268818] LNet: Removed LNI 192.168.201.16@tcp [ 646.332523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 646.423084] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 646.458115] LNet: Added LNI 192.168.201.16@tcp [8/256/0/180] [ 647.021879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.201.[17-22/2]@tcp [ 647.052600] LNet: 31597:0:(router.c:728:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 647.060989] LNet: 31597:0:(router.c:728:lnet_add_route()) Skipped 4 previous similar messages [ 647.084668] LNetError: 31484:0:(lib-lnet.h:1336:lnet_set_route_aliveness()) route to tcp103 through 192.168.201.17@tcp has gone from up to down [ 647.095892] LNetError: 31484:0:(lib-lnet.h:1336:lnet_set_route_aliveness()) Skipped 1 previous similar message [ 647.096130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.201.[17-22/2]@tcp [ 655.188902] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 20:35:35 (1778286935) [ 655.327158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 655.387686] LNetError: 32187:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 655.436652] LNet: Removed LNI 192.168.201.16@tcp [ 655.483256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 655.588558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 655.667261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 655.723546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 655.822467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 655.893498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 655.926271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 655.999591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 656.089264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 656.125244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 656.196213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 656.325324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 656.395074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 656.490227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 656.612794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 656.658779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 656.784090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 656.946790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 657.016636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 657.120454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 657.301777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 657.351090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 657.432477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 657.539790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 657.593589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 657.649922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 657.726563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 657.788667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 657.895632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 658.003292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 658.027971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 658.116561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 658.173182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 658.218609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 658.279681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 664.065431] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 20:35:45 (1778286945) [ 665.993663] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg116-server [ 667.674508] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 20:35:49 (1778286949) [ 669.516378] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg116-server [ 671.064723] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 20:35:52 (1778286952) [ 671.797402] Key type .llcrypt unregistered [ 671.799352] Key type ._llcrypt unregistered [ 677.376863] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 677.464236] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 677.481131] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 677.491193] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 680.118463] Key type ._llcrypt registered [ 680.122310] Key type .llcrypt registered [ 680.296453] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 680.319748] alg: No test for adler32 (adler32-zlib) [ 681.218976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 681.271647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 681.307320] LNet: Added LNI 192.168.201.16@tcp [8/256/0/180] [ 681.908728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 681.928825] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 684.082752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 684.096411] LNet: Removed LNI 10.1.2.3@tcp [ 684.236411] LNet: 34316:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 684.241343] LNetError: 34316:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 684.257463] LNet: Removed LNI 192.168.201.16@tcp [ 684.885298] Key type .llcrypt unregistered [ 684.887370] Key type ._llcrypt unregistered [ 696.640241] Key type ._llcrypt registered [ 696.642526] Key type .llcrypt registered [ 697.917250] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 20:36:19 (1778286979) [ 698.552339] Key type .llcrypt unregistered [ 698.557594] Key type ._llcrypt unregistered [ 704.119846] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 704.330753] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 704.339607] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 704.342170] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 706.801929] Key type ._llcrypt registered [ 706.804218] Key type .llcrypt registered [ 706.969508] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 706.982829] alg: No test for adler32 (adler32-zlib) [ 707.929399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 707.971481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 708.001384] LNet: Added LNI 192.168.201.16@tcp [8/256/0/180] [ 708.857256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 708.886180] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 711.704417] LNet: 35808:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 711.711692] LNetError: 35808:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 711.722296] LNet: Removed LNI 192.168.201.16@tcp [ 711.740380] LNet: Removed LNI 10.1.2.3@tcp [ 712.351534] Key type .llcrypt unregistered [ 712.354094] Key type ._llcrypt unregistered [ 723.197754] Key type ._llcrypt registered [ 723.203408] Key type .llcrypt registered [ 724.893773] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 20:36:45 (1778287005) [ 725.831416] Key type .llcrypt unregistered [ 725.836493] Key type ._llcrypt unregistered [ 732.526816] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 732.711159] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 732.716976] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 732.724260] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 735.305899] Key type ._llcrypt registered [ 735.309371] Key type .llcrypt registered [ 735.514556] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 735.537467] alg: No test for adler32 (adler32-zlib) [ 736.539592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 736.622755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 736.659947] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 738.932614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 738.961414] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 741.216926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 741.235657] LNet: Removed LNI 10.1.2.3@tcp [ 741.246107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 741.267503] LNet: Removed LNI 10.1.2.31@tcp [ 741.290763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 741.311075] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 742.374935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 742.409082] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 743.439932] LNet: 37729:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 743.445594] LNetError: 37729:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 743.457186] LNet: Removed LNI 10.1.2.3@tcp [ 743.508483] LNet: Removed LNI 10.1.2.31@tcp [ 744.346475] Key type .llcrypt unregistered [ 744.348315] Key type ._llcrypt unregistered [ 768.247544] Key type ._llcrypt registered [ 768.249569] Key type .llcrypt registered [ 772.241620] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 20:37:30 (1778287050) [ 772.569506] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 772.583731] alg: No test for adler32 (adler32-zlib) [ 773.651162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 773.808579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 773.872924] LNet: Added LNI 192.168.201.16@tcp [8/256/0/180] [ 775.804245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 775.821647] LNet: 38684:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 775.827739] LNetError: 38684:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 775.839783] LNet: Removed LNI 192.168.201.16@tcp [ 775.852493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 775.892754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.201.16@tcp [ 775.910032] LNet: Added LNI 192.168.201.16@tcp [8/256/0/180] [ 777.083519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.201.16@tcp [ 777.115576] LNet: Removed LNI 192.168.201.16@tcp [ 777.216568] LNet: 38820:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 778.191823] Key type .llcrypt unregistered [ 778.195766] Key type ._llcrypt unregistered [ 794.407340] Key type ._llcrypt registered [ 794.409475] Key type .llcrypt registered [ 796.613343] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 20:37:56 (1778287076) [ 799.023131] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg116-server [ 801.309931] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 20:38:02 (1778287082) [ 802.435312] Key type .llcrypt unregistered [ 802.439227] Key type ._llcrypt unregistered [ 809.028698] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 809.210818] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 809.222821] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 809.225293] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 812.002082] Key type ._llcrypt registered [ 812.006276] Key type .llcrypt registered [ 812.240276] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 812.254715] alg: No test for adler32 (adler32-zlib) [ 813.272345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 813.404540] LNet: Added LNI 192.168.201.16@tcp1 [8/256/0/180] [ 814.631788] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 817.757620] LNet: 40653:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 817.766459] LNetError: 40653:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 817.777091] LNet: Removed LNI 192.168.201.16@tcp1 [ 817.804878] LNet: Removed LNI 10.1.2.3@tcp2 [ 818.585760] Key type .llcrypt unregistered [ 818.590383] Key type ._llcrypt unregistered [ 831.087075] Key type ._llcrypt registered [ 831.097269] Key type .llcrypt registered [ 832.517858] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 20:38:33 (1778287113) [ 833.038581] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 833.046535] alg: No test for adler32 (adler32-zlib) [ 834.000709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 834.108583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 834.150652] LNet: Added LNI 192.168.201.16@tcp [8/256/0/180] [ 835.121882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 835.146968] LNet: 41622:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 835.160378] LNetError: 41622:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 835.180593] LNet: Removed LNI 192.168.201.16@tcp [ 835.214083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 835.304569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 835.342075] LNet: Added LNI 192.168.201.16@tcp [8/256/0/180] [ 843.220669] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 20:38:44 (1778287124) [ 843.360419] LNet: 42208:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 843.371968] LNetError: 42208:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 843.406659] LNet: Removed LNI 192.168.201.16@tcp [ 844.316984] Key type .llcrypt unregistered [ 844.319474] Key type ._llcrypt unregistered [ 851.513881] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 851.685590] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 851.691125] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 851.697052] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 854.198958] Key type ._llcrypt registered [ 854.202491] Key type .llcrypt registered [ 854.429902] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 854.455719] alg: No test for adler32 (adler32-zlib) [ 855.416834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 855.540443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 855.587610] LNet: Added LNI 192.168.201.16@tcp [8/256/0/180] [ 856.534316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 856.561044] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 859.190420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 859.233251] LNet: 42957:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 859.243503] LNetError: 42957:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 859.257976] LNet: Removed LNI 192.168.201.16@tcp [ 859.277298] LNet: Removed LNI 10.1.2.3@tcp [ 859.338283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 859.454177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 859.485496] LNet: Added LNI 192.168.201.16@tcp [8/256/0/180] [ 859.494485] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 860.181856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 860.220626] LNet: 42984:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 860.244224] LNetError: 42984:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 860.254978] LNetError: 42984:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 860.266752] LNet: Removed LNI 192.168.201.16@tcp [ 860.304504] LNet: Removed LNI 10.1.2.3@tcp [ 860.340904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 860.416398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 860.449314] LNet: Added LNI 192.168.201.16@tcp [8/256/0/180] [ 860.470308] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 860.981631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 861.006851] LNet: Added LNI 192.168.201.16@tcp2 [8/256/0/180] [ 862.879131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 862.916968] LNet: 43147:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 862.930904] LNetError: 43147:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 862.938700] LNetError: 43147:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 862.949115] LNet: Removed LNI 192.168.201.16@tcp [ 862.951139] LNet: Removed LNI 10.1.2.3@tcp [ 862.979735] LNet: Removed LNI 192.168.201.16@tcp2 [ 863.018996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 863.157742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 863.218613] LNet: Added LNI 192.168.201.16@tcp [8/256/0/180] [ 863.221354] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 863.239238] LNet: Added LNI 192.168.201.16@tcp2 [8/256/0/180] [ 871.786539] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 20:39:12 (1778287152) [ 871.902707] LNet: 43735:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 871.919731] LNetError: 43735:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 871.943985] LNetError: 43735:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 871.953032] LNet: Removed LNI 192.168.201.16@tcp [ 871.954913] LNet: Removed LNI 10.1.2.3@tcp [ 871.999107] LNet: Removed LNI 192.168.201.16@tcp2 [ 872.709979] Key type .llcrypt unregistered [ 872.718201] Key type ._llcrypt unregistered [ 880.105851] Key type ._llcrypt registered [ 880.108073] Key type .llcrypt registered [ 880.445296] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 880.477466] alg: No test for adler32 (adler32-zlib) [ 881.389723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 881.485765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 881.532163] LNet: Added LNI 192.168.201.16@tcp [8/256/0/180] [ 882.658985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 882.697948] LNet: 44139:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 882.709521] LNetError: 44139:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 882.724798] LNet: Removed LNI 192.168.201.16@tcp [ 882.766737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 882.838080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 882.856221] LNet: Added LNI 192.168.201.16@tcp [8/256/0/180] [ 883.327375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 883.359543] LNet: 44169:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 883.375139] LNetError: 44169:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 883.389467] LNet: Removed LNI 192.168.201.16@tcp [ 883.413265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 883.531190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 883.552206] LNet: Added LNI 192.168.201.16@tcp [8/256/0/180] [ 883.987297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 884.039322] LNet: Removed LNI 192.168.201.16@tcp [ 884.060595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 884.157456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 884.203982] LNet: Added LNI 192.168.201.16@tcp [8/256/0/180] [ 884.830092] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 884.998555] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 885.007614] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 885.009924] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 887.413958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 887.459926] LNet: 44280:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 887.468060] LNet: 44280:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 887.491157] LNetError: 44280:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 887.500567] LNetError: 44280:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 887.530948] LNet: Removed LNI 192.168.201.16@tcp [ 887.569720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 887.694344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 887.724959] LNet: Added LNI 192.168.201.16@tcp [8/256/0/180] [ 894.695330] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 20:39:35 (1778287175) [ 894.865499] LNet: 44876:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 894.884346] LNetError: 44876:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 894.909661] LNet: Removed LNI 192.168.201.16@tcp [ 895.490176] Key type .llcrypt unregistered [ 895.491721] Key type ._llcrypt unregistered [ 902.371958] Key type ._llcrypt registered [ 902.383479] Key type .llcrypt registered [ 902.682453] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 902.724320] alg: No test for adler32 (adler32-zlib) [ 903.678578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 903.728938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 903.746258] LNet: Added LNI 192.168.201.16@tcp [8/256/0/180] [ 904.772598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 904.803725] LNet: 45276:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 904.814541] LNetError: 45276:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 904.833176] LNet: Removed LNI 192.168.201.16@tcp [ 904.884573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 904.977658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 905.033048] LNet: Added LNI 192.168.201.16@tcp [8/256/0/180] [ 912.924706] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 20:39:54 (1778287194) [ 913.078636] LNet: 45867:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 913.091602] LNetError: 45867:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 913.137983] LNet: Removed LNI 192.168.201.16@tcp [ 913.725536] Key type .llcrypt unregistered [ 913.729170] Key type ._llcrypt unregistered [ 920.190989] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 920.472875] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 920.491768] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 920.496493] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 923.169988] Key type ._llcrypt registered [ 923.176258] Key type .llcrypt registered [ 923.405815] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 923.428108] alg: No test for adler32 (adler32-zlib) [ 924.415313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 924.504479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 924.534127] LNet: Added LNI 192.168.201.16@tcp [8/256/0/180] [ 925.796747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 925.818945] LNet: 46320:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 925.826074] LNetError: 46320:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 925.850258] LNet: Removed LNI 192.168.201.16@tcp [ 925.872416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 925.940302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 925.977178] LNet: Added LNI 192.168.201.16@tcp [8/256/0/180] [ 934.027742] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 20:40:15 (1778287215) [ 934.193134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 934.230704] LNet: 46914:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 934.249226] LNetError: 46914:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 934.282388] LNet: Removed LNI 192.168.201.16@tcp [ 934.319476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 934.406315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 940.902807] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 20:40:22 (1778287222) [ 941.218669] LNet: 47500:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 941.873378] Key type .llcrypt unregistered [ 941.875886] Key type ._llcrypt unregistered [ 949.313353] Key type ._llcrypt registered [ 949.318815] Key type .llcrypt registered [ 949.618699] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 949.638686] alg: No test for adler32 (adler32-zlib) [ 950.668137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 950.756316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 950.787859] LNet: Added LNI 192.168.201.16@tcp [8/256/0/180] [ 951.794091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 951.810822] LNet: 47898:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 951.819513] LNetError: 47898:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 951.836929] LNet: Removed LNI 192.168.201.16@tcp [ 951.864666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 951.910568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 951.928728] LNet: Added LNI 192.168.201.16@tcp [8/256/0/180] [ 958.645525] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 20:40:39 (1778287239) [ 958.987716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 959.011438] LNet: 48491:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 959.020465] LNetError: 48491:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 959.034370] LNet: Removed LNI 192.168.201.16@tcp [ 959.059535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 959.117373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 965.375434] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 20:40:46 (1778287246) [ 966.937959] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 968.774302] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 20:40:50 (1778287250) [ 970.250067] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 972.178897] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 20:40:53 (1778287253) [ 973.942213] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 976.397427] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 20:40:57 (1778287257) [ 978.452235] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 980.024993] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 20:41:01 (1778287261) [ 981.495811] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 983.408660] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 20:41:04 (1778287264) [ 983.637823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 983.653329] LNet: 50543:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 983.670563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 983.740924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 983.783064] LNet: Added LNI 192.168.201.16@tcp [8/256/0/180] [ 984.575289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.201.16 [ 984.597940] LNet: 50676:0:(config.c:1608:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 984.608393] LNet: Removed LNI 192.168.201.16@tcp [ 990.885785] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 20:41:12 (1778287272) [ 991.127073] LNet: 51247:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 991.137606] LNetError: 51247:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 991.163579] LNet: Removed LNI 192.168.201.16@tcp [ 991.957329] Key type .llcrypt unregistered [ 991.960389] Key type ._llcrypt unregistered [ 997.832710] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 998.026665] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 998.030078] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 998.035499] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1000.901244] Key type ._llcrypt registered [ 1000.903853] Key type .llcrypt registered [ 1001.028187] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1001.041529] alg: No test for adler32 (adler32-zlib) [ 1002.653958] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1009.345860] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 20:41:30 (1778287290) [ 1009.787736] LNet: 52151:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1009.800152] LNetError: 52151:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1009.820275] LNet: Removed LNI 10.1.2.3@tcp [ 1010.494303] Key type .llcrypt unregistered [ 1010.496429] Key type ._llcrypt unregistered [ 1017.682236] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1017.945610] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1017.956455] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1017.964297] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1019.644659] Key type ._llcrypt registered [ 1019.649667] Key type .llcrypt registered [ 1019.844960] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1019.863405] alg: No test for adler32 (adler32-zlib) [ 1020.864443] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1027.111475] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 20:41:48 (1778287308) [ 1027.239388] LNet: 53028:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1027.253276] LNetError: 53028:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1027.274019] LNet: Removed LNI 10.1.2.3@tcp [ 1028.073922] Key type .llcrypt unregistered [ 1028.078887] Key type ._llcrypt unregistered [ 1035.984079] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1036.284179] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1036.295104] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1036.308908] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1039.143307] Key type ._llcrypt registered [ 1039.146563] Key type .llcrypt registered [ 1039.398716] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1039.413251] alg: No test for adler32 (adler32-zlib) [ 1040.341667] LNet: 53338:0:(config.c:1608:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1040.364070] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1046.698847] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 20:42:07 (1778287327) [ 1046.887523] LNet: 53924:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1046.905693] LNetError: 53924:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1046.946699] LNet: Removed LNI 10.1.2.3@tcp [ 1047.586759] Key type .llcrypt unregistered [ 1047.589503] Key type ._llcrypt unregistered [ 1054.391776] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1054.659797] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1054.663786] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1054.673843] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1057.529322] Key type ._llcrypt registered [ 1057.532800] Key type .llcrypt registered [ 1057.718069] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1057.733848] alg: No test for adler32 (adler32-zlib) [ 1058.654672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1058.704776] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1068.017508] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 20:42:29 (1778287349) [ 1068.820778] LNet: 55159:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1068.830087] LNetError: 55159:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1068.853748] LNet: Removed LNI 10.1.2.3@tcp [ 1069.560702] Key type .llcrypt unregistered [ 1069.562270] Key type ._llcrypt unregistered [ 1075.728899] Key type ._llcrypt registered [ 1075.731861] Key type .llcrypt registered [ 1075.914960] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1075.933545] alg: No test for adler32 (adler32-zlib) [ 1076.897696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1076.953329] LNet: Added LNI 192.168.201.16@tcp [8/256/0/180] [ 1085.480507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.116@tcp [ 1086.346394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.16@tcp -d 192.168.201.116@tcp -m GET -e local_interrupt -r 1 [ 1086.395876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.16@tcp -d 192.168.201.16@tcp -m GET -e local_interrupt -r 1 [ 1086.442508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.116@tcp [ 1086.463468] LNet: There was an unexpected network error while writing to 192.168.201.116: rc = -22 [ 1086.816219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1086.864977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1086.920552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1087.002412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1087.074052] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1087.118919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1087.196302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1087.203931] LNet: 1 local NIs in recovery (showing 1): 192.168.201.16@tcp [ 1087.249744] LNet: There was an unexpected network error while writing to 192.168.201.16: rc = -22 [ 1087.333866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1087.392669] LNet: 55597:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1087.454633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1087.538722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1087.591502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1087.652478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1087.701181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1087.761291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1087.819388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1088.434454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.16@tcp -d 192.168.201.116@tcp -m GET -e local_dropped -r 1 [ 1088.468203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.16@tcp -d 192.168.201.16@tcp -m GET -e local_dropped -r 1 [ 1088.517552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.116@tcp [ 1088.533556] LNet: There was an unexpected network error while writing to 192.168.201.116: rc = -22 [ 1088.763829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1088.789722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1088.812830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1088.843539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1088.876481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1088.906903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1088.939064] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1088.983404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1089.002440] LNet: 55706:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1089.023554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1089.066422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1089.100856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1089.123220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1089.155551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1089.187217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1089.213575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1089.725337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.16@tcp -d 192.168.201.116@tcp -m GET -e local_aborted -r 1 [ 1089.777619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.16@tcp -d 192.168.201.16@tcp -m GET -e local_aborted -r 1 [ 1089.822896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.116@tcp [ 1090.044808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1090.078347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1090.118604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1090.164316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1090.203056] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1090.227090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1090.258404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1090.289440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1090.307925] LNet: 55815:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1090.333030] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1090.372872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1090.411357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1090.438319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1090.465604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1090.494359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1090.520954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1091.044587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.16@tcp -d 192.168.201.116@tcp -m GET -e local_no_route -r 1 [ 1091.075645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.16@tcp -d 192.168.201.16@tcp -m GET -e local_no_route -r 1 [ 1091.105219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.116@tcp [ 1091.123486] LNet: There was an unexpected network error while writing to 192.168.201.116: rc = -22 [ 1091.127798] LNet: Skipped 1 previous similar message [ 1091.291211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1091.317316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1091.349629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1091.380341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1091.408315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1091.434828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1091.462289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1091.500928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1091.543970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1091.585359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1091.623756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1091.668710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1091.694955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1091.769069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1091.830927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1092.525049] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.16@tcp -d 192.168.201.116@tcp -m GET -e local_timeout -r 1 [ 1092.600223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.16@tcp -d 192.168.201.16@tcp -m GET -e local_timeout -r 1 [ 1092.647930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.116@tcp [ 1092.951885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1092.984878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1093.022080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1093.052233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1093.077165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1093.104238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1093.140417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1093.174909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1093.198263] LNet: 56034:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1093.204293] LNet: 56034:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1093.216956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1093.241351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1093.265204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1093.283946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1093.305554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1093.329438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1093.350246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1093.893265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.16@tcp -d 192.168.201.116@tcp -m GET -e local_error -r 1 [ 1093.922568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.16@tcp -d 192.168.201.16@tcp -m GET -e local_error -r 1 [ 1093.979883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.116@tcp [ 1094.260274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1094.307501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1094.395416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1094.427326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1094.452727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1094.478076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1094.504507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1094.534319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1094.561511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1094.594363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1094.629672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1094.657148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1094.688813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1094.714246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1094.739606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1094.980687] LNet: 56184:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1094.987967] LNetError: 56184:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1095.014328] LNet: Removed LNI 192.168.201.16@tcp [ 1095.400678] Key type .llcrypt unregistered [ 1095.408268] Key type ._llcrypt unregistered [ 1110.608231] Key type ._llcrypt registered [ 1110.610728] Key type .llcrypt registered [ 1111.955188] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 20:43:13 (1778287393) [ 1112.803806] Key type .llcrypt unregistered [ 1112.809761] Key type ._llcrypt unregistered [ 1119.542049] Key type ._llcrypt registered [ 1119.545026] Key type .llcrypt registered [ 1119.897869] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1119.929658] alg: No test for adler32 (adler32-zlib) [ 1120.806668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1120.827515] LNet: Added LNI 192.168.201.16@tcp [8/256/0/180] [ 1128.705864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.116@tcp [ 1139.109708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1139.162551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1139.222822] LNet: Added LNI 192.168.201.16@tcp1 [8/256/0/180] [ 1140.959338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.16@tcp -d 192.168.201.116@tcp -m GET -e local_interrupt -r 1 [ 1140.998591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.16@tcp -d 192.168.201.116@tcp1 -m GET -e local_interrupt -r 1 [ 1141.034609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.16@tcp -d 192.168.201.16@tcp -m GET -e local_interrupt -r 1 [ 1141.076363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.16@tcp -d 192.168.201.16@tcp1 -m GET -e local_interrupt -r 1 [ 1141.132424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.16@tcp1 -d 192.168.201.116@tcp -m GET -e local_interrupt -r 1 [ 1141.175270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.16@tcp1 -d 192.168.201.116@tcp1 -m GET -e local_interrupt -r 1 [ 1141.226366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.16@tcp1 -d 192.168.201.16@tcp -m GET -e local_interrupt -r 1 [ 1141.265966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.16@tcp1 -d 192.168.201.16@tcp1 -m GET -e local_interrupt -r 1 [ 1141.326378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.116@tcp [ 1141.372552] LNet: There was an unexpected network error while writing to 192.168.201.116: rc = -22 [ 1141.377853] LNet: 1 local NIs in recovery (showing 1): 192.168.201.16@tcp [ 1141.694468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1141.751546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1141.843763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1141.894035] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1141.975678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1142.022900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1142.073459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1142.119669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1142.139937] LNet: 57649:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1142.164668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1142.202489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1142.232307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1142.259966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1142.288275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1142.326658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1142.363098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1143.007081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.16@tcp -d 192.168.201.116@tcp -m GET -e local_dropped -r 1 [ 1143.046861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.16@tcp -d 192.168.201.116@tcp1 -m GET -e local_dropped -r 1 [ 1143.085578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.16@tcp -d 192.168.201.16@tcp -m GET -e local_dropped -r 1 [ 1143.132090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.16@tcp -d 192.168.201.16@tcp1 -m GET -e local_dropped -r 1 [ 1143.189196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.16@tcp1 -d 192.168.201.116@tcp -m GET -e local_dropped -r 1 [ 1143.229685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.16@tcp1 -d 192.168.201.116@tcp1 -m GET -e local_dropped -r 1 [ 1143.286510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.16@tcp1 -d 192.168.201.16@tcp -m GET -e local_dropped -r 1 [ 1143.331525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.16@tcp1 -d 192.168.201.16@tcp1 -m GET -e local_dropped -r 1 [ 1143.389198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.116@tcp [ 1143.447072] LNet: There was an unexpected network error while writing to 192.168.201.116: rc = -22 [ 1143.450195] LNet: Skipped 2 previous similar messages [ 1143.755150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1143.788671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1143.813226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1143.834752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1143.860380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1143.888156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1143.912922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1143.934417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1143.949227] LNet: 57771:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1143.963379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1143.989405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1144.018978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1144.041614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1144.068894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1144.100300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1144.140897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1144.734285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.16@tcp -d 192.168.201.116@tcp -m GET -e local_aborted -r 1 [ 1144.768308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.16@tcp -d 192.168.201.116@tcp1 -m GET -e local_aborted -r 1 [ 1144.807343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.16@tcp -d 192.168.201.16@tcp -m GET -e local_aborted -r 1 [ 1144.858500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.16@tcp -d 192.168.201.16@tcp1 -m GET -e local_aborted -r 1 [ 1144.897150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.16@tcp1 -d 192.168.201.116@tcp -m GET -e local_aborted -r 1 [ 1144.954823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.16@tcp1 -d 192.168.201.116@tcp1 -m GET -e local_aborted -r 1 [ 1145.004471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.16@tcp1 -d 192.168.201.16@tcp -m GET -e local_aborted -r 1 [ 1145.051970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.16@tcp1 -d 192.168.201.16@tcp1 -m GET -e local_aborted -r 1 [ 1145.110372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.116@tcp [ 1145.134163] LNet: There was an unexpected network error while writing to 192.168.201.116: rc = -22 [ 1145.139133] LNet: Skipped 2 previous similar messages [ 1145.410389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1145.456674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1145.484396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1145.519116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1145.553546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1145.599574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1145.629180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1145.653903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1145.672466] LNet: 57892:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1145.700441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1145.728269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1145.765400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1145.799883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1145.857381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1145.909466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1145.960223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1146.561581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.16@tcp -d 192.168.201.116@tcp -m GET -e local_no_route -r 1 [ 1146.602538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.16@tcp -d 192.168.201.116@tcp1 -m GET -e local_no_route -r 1 [ 1146.647335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.16@tcp -d 192.168.201.16@tcp -m GET -e local_no_route -r 1 [ 1146.686908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.16@tcp -d 192.168.201.16@tcp1 -m GET -e local_no_route -r 1 [ 1146.719205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.16@tcp1 -d 192.168.201.116@tcp -m GET -e local_no_route -r 1 [ 1146.749383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.16@tcp1 -d 192.168.201.116@tcp1 -m GET -e local_no_route -r 1 [ 1146.786204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.16@tcp1 -d 192.168.201.16@tcp -m GET -e local_no_route -r 1 [ 1146.817604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.16@tcp1 -d 192.168.201.16@tcp1 -m GET -e local_no_route -r 1 [ 1146.850837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.116@tcp [ 1147.320241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1147.390605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1147.472617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1147.523679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1147.581988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1147.631101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1147.679554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1147.731429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1147.751238] LNet: 58013:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1147.769527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1147.812395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1147.857593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1147.883724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1147.936618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1147.938545] LNet: There was an unexpected network error while writing to 192.168.201.16: rc = -22 [ 1147.960868] LNet: Skipped 6 previous similar messages [ 1147.984713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1148.034985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1148.708884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.16@tcp -d 192.168.201.116@tcp -m GET -e local_timeout -r 1 [ 1148.735597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.16@tcp -d 192.168.201.116@tcp1 -m GET -e local_timeout -r 1 [ 1148.781864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.16@tcp -d 192.168.201.16@tcp -m GET -e local_timeout -r 1 [ 1148.819192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.16@tcp -d 192.168.201.16@tcp1 -m GET -e local_timeout -r 1 [ 1148.866896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.16@tcp1 -d 192.168.201.116@tcp -m GET -e local_timeout -r 1 [ 1148.941414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.16@tcp1 -d 192.168.201.116@tcp1 -m GET -e local_timeout -r 1 [ 1148.996313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.16@tcp1 -d 192.168.201.16@tcp -m GET -e local_timeout -r 1 [ 1149.034991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.16@tcp1 -d 192.168.201.16@tcp1 -m GET -e local_timeout -r 1 [ 1149.084461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.116@tcp [ 1149.417493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1149.472960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1149.517743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1149.558720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1149.599562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1149.640716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1149.691561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1149.728351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1149.762281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1149.799065] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1149.854322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1149.892084] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1149.933993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1149.970563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1150.013516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1150.490733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.16@tcp -d 192.168.201.116@tcp -m GET -e local_error -r 1 [ 1150.550061] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.16@tcp -d 192.168.201.116@tcp1 -m GET -e local_error -r 1 [ 1150.596836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.16@tcp -d 192.168.201.16@tcp -m GET -e local_error -r 1 [ 1150.653887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.16@tcp -d 192.168.201.16@tcp1 -m GET -e local_error -r 1 [ 1150.683964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.16@tcp1 -d 192.168.201.116@tcp -m GET -e local_error -r 1 [ 1150.710641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.16@tcp1 -d 192.168.201.116@tcp1 -m GET -e local_error -r 1 [ 1150.740515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.16@tcp1 -d 192.168.201.16@tcp -m GET -e local_error -r 1 [ 1150.770166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.16@tcp1 -d 192.168.201.16@tcp1 -m GET -e local_error -r 1 [ 1150.805528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.116@tcp [ 1151.150449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1151.191568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1151.234420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1151.268216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1151.287208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1151.322619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1151.345939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1151.370243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1151.401687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1151.431291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1151.464763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1151.512483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1151.552032] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1151.598656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1151.646870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1155.864623] LNet: 58332:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1155.882958] LNetError: 58332:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1155.892787] LNet: Removed LNI 192.168.201.16@tcp [ 1156.965871] LNet: Removed LNI 192.168.201.16@tcp1 [ 1157.425803] Key type .llcrypt unregistered [ 1157.431039] Key type ._llcrypt unregistered [ 1176.470996] Key type ._llcrypt registered [ 1176.472673] Key type .llcrypt registered [ 1178.009703] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 20:44:18 (1778287458) [ 1178.685957] Key type .llcrypt unregistered [ 1178.691422] Key type ._llcrypt unregistered [ 1185.416048] Key type ._llcrypt registered [ 1185.419584] Key type .llcrypt registered [ 1185.730610] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1185.755187] alg: No test for adler32 (adler32-zlib) [ 1186.731912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1186.779563] LNet: Added LNI 192.168.201.16@tcp [8/256/0/180] [ 1194.101696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.116@tcp [ 1194.805587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.16@tcp -d 192.168.201.116@tcp -m GET -e remote_dropped -r 1 [ 1194.846582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.16@tcp -d 192.168.201.16@tcp -m GET -e remote_dropped -r 1 [ 1194.900339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.116@tcp [ 1194.920526] LNet: There was an unexpected network error while writing to 192.168.201.116: rc = -22 [ 1195.171335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1195.213938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1195.260426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1195.281893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1195.312800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1195.339985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1195.365648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1195.400537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1195.417266] LNet: 59580:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1195.435404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1195.470806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1195.516276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1195.548916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1195.588053] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1195.630799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1195.658817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1196.251317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.16@tcp -d 192.168.201.116@tcp -m GET -e remote_error -r 1 [ 1196.313695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.16@tcp -d 192.168.201.16@tcp -m GET -e remote_error -r 1 [ 1196.358809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.116@tcp [ 1196.391783] LNet: There was an unexpected network error while writing to 192.168.201.116: rc = -22 [ 1196.603516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1196.637170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1196.666188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1196.703213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1196.730264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1196.773964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1196.809897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1196.840183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1196.862459] LNet: 59689:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1196.892742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1196.915297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1196.949652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1196.997058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1197.024825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1197.057397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1197.094397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1197.549194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.16@tcp -d 192.168.201.116@tcp -m GET -e remote_timeout -r 1 [ 1197.577408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.16@tcp -d 192.168.201.16@tcp -m GET -e remote_timeout -r 1 [ 1197.608616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.116@tcp [ 1197.626082] LNet: There was an unexpected network error while writing to 192.168.201.116: rc = -22 [ 1197.868806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1197.899597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1197.932595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1198.002899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1198.060455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1198.119535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1198.163248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1198.197642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1198.212557] LNet: 59799:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1198.230325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1198.253794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1198.279982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1198.304584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1198.323930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1198.351827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1198.400461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1198.600202] LNet: 59840:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1198.605627] LNetError: 59840:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1198.629344] LNet: Removed LNI 192.168.201.16@tcp [ 1199.029720] Key type .llcrypt unregistered [ 1199.035629] Key type ._llcrypt unregistered [ 1215.428879] Key type ._llcrypt registered [ 1215.431783] Key type .llcrypt registered [ 1216.968194] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 20:44:58 (1778287498) [ 1217.819360] Key type .llcrypt unregistered [ 1217.821362] Key type ._llcrypt unregistered [ 1224.796254] Key type ._llcrypt registered [ 1224.797974] Key type .llcrypt registered [ 1225.142576] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1225.160220] alg: No test for adler32 (adler32-zlib) [ 1226.140613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1226.180448] LNet: Added LNI 192.168.201.16@tcp [8/256/0/180] [ 1234.179128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.116@tcp [ 1244.142766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1244.227452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1244.256459] LNet: Added LNI 192.168.201.16@tcp1 [8/256/0/180] [ 1245.931852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.16@tcp -d 192.168.201.116@tcp -m GET -e remote_dropped -r 1 [ 1245.982809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.16@tcp -d 192.168.201.116@tcp1 -m GET -e remote_dropped -r 1 [ 1246.033716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.16@tcp -d 192.168.201.16@tcp -m GET -e remote_dropped -r 1 [ 1246.086804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.16@tcp -d 192.168.201.16@tcp1 -m GET -e remote_dropped -r 1 [ 1246.127664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.16@tcp1 -d 192.168.201.116@tcp -m GET -e remote_dropped -r 1 [ 1246.173584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.16@tcp1 -d 192.168.201.116@tcp1 -m GET -e remote_dropped -r 1 [ 1246.214462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.16@tcp1 -d 192.168.201.16@tcp -m GET -e remote_dropped -r 1 [ 1246.281334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.16@tcp1 -d 192.168.201.16@tcp1 -m GET -e remote_dropped -r 1 [ 1246.374652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.116@tcp [ 1246.417233] LNet: There was an unexpected network error while writing to 192.168.201.116: rc = -22 [ 1246.435327] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.116@tcp [ 1246.751115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1246.787837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1246.819650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1246.858533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1246.897779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1246.940588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1246.983558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1247.043535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1247.084430] LNet: 61307:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1247.110789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1247.180626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1247.247978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1247.307582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1247.335285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1247.380823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1247.434483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1247.478331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1248.191521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.16@tcp -d 192.168.201.116@tcp -m GET -e remote_error -r 1 [ 1248.278760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.16@tcp -d 192.168.201.116@tcp1 -m GET -e remote_error -r 1 [ 1248.378615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.16@tcp -d 192.168.201.16@tcp -m GET -e remote_error -r 1 [ 1248.488216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.16@tcp -d 192.168.201.16@tcp1 -m GET -e remote_error -r 1 [ 1248.556521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.16@tcp1 -d 192.168.201.116@tcp -m GET -e remote_error -r 1 [ 1248.626091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.16@tcp1 -d 192.168.201.116@tcp1 -m GET -e remote_error -r 1 [ 1248.730398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.16@tcp1 -d 192.168.201.16@tcp -m GET -e remote_error -r 1 [ 1248.842352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.16@tcp1 -d 192.168.201.16@tcp1 -m GET -e remote_error -r 1 [ 1248.955081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.116@tcp [ 1249.006390] LNet: There was an unexpected network error while writing to 192.168.201.116: rc = -22 [ 1249.017580] LNet: Skipped 2 previous similar messages [ 1249.369365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1249.433243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1249.503757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1249.561543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1249.622367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1249.703593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1249.770765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1249.833851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1249.873595] LNet: 61430:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1249.900642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1249.932224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1249.992930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1250.032242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1250.053351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1250.084345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1250.112333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1250.139171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1250.741037] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.16@tcp -d 192.168.201.116@tcp -m GET -e remote_timeout -r 1 [ 1250.792945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.16@tcp -d 192.168.201.116@tcp1 -m GET -e remote_timeout -r 1 [ 1250.857974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.16@tcp -d 192.168.201.16@tcp -m GET -e remote_timeout -r 1 [ 1250.912821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.16@tcp -d 192.168.201.16@tcp1 -m GET -e remote_timeout -r 1 [ 1250.961835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.16@tcp1 -d 192.168.201.116@tcp -m GET -e remote_timeout -r 1 [ 1251.016715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.16@tcp1 -d 192.168.201.116@tcp1 -m GET -e remote_timeout -r 1 [ 1251.058945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.16@tcp1 -d 192.168.201.16@tcp -m GET -e remote_timeout -r 1 [ 1251.113637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.16@tcp1 -d 192.168.201.16@tcp1 -m GET -e remote_timeout -r 1 [ 1251.162961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.116@tcp [ 1251.204797] LNet: There was an unexpected network error while writing to 192.168.201.116: rc = -22 [ 1251.541356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1251.589859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1251.639844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1251.703594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1251.762632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1251.814321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1251.897746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1251.973489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1251.995128] LNet: 61554:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1252.010976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1252.037132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1252.067429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1252.097734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1252.162104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1252.265252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1252.311210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1252.382400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1257.336968] LNet: 61632:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1257.347449] LNetError: 61632:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1257.357118] LNet: Removed LNI 192.168.201.16@tcp [ 1257.372949] LNet: Removed LNI 192.168.201.16@tcp1 [ 1258.176793] Key type .llcrypt unregistered [ 1258.178393] Key type ._llcrypt unregistered [ 1281.570719] Key type ._llcrypt registered [ 1281.572513] Key type .llcrypt registered [ 1283.153825] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 20:46:04 (1778287564) [ 1283.889328] Key type .llcrypt unregistered [ 1283.891483] Key type ._llcrypt unregistered [ 1291.158509] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1291.328866] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1291.343668] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1291.364356] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1294.270029] Key type ._llcrypt registered [ 1294.273348] Key type .llcrypt registered [ 1294.480568] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1294.504714] alg: No test for adler32 (adler32-zlib) [ 1295.557494] LNet: 62713:0:(config.c:1608:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1295.582436] LNet: Added LNI 192.168.201.16@tcp [8/256/0/180] [ 1295.653064] LNet: 62727:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1295.662784] LNetError: 62727:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1295.684181] LNet: Removed LNI 192.168.201.16@tcp [ 1296.491496] Key type .llcrypt unregistered [ 1296.495357] Key type ._llcrypt unregistered [ 1303.494860] Key type ._llcrypt registered [ 1303.496483] Key type .llcrypt registered [ 1303.669256] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1303.684352] alg: No test for adler32 (adler32-zlib) [ 1304.715520] LNet: 62986:0:(config.c:1608:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1304.747803] LNet: Added LNI 192.168.201.16@tcp [8/256/0/180] [ 1304.751920] LNet: Added LNI 10.1.2.3@tcp1 [8/256/0/180] [ 1304.788734] LNet: 63000:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1304.797520] LNetError: 63000:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1304.806847] LNet: Removed LNI 192.168.201.16@tcp [ 1304.818854] LNet: Removed LNI 10.1.2.3@tcp1 [ 1305.572619] Key type .llcrypt unregistered [ 1305.575638] Key type ._llcrypt unregistered [ 1312.371909] Key type ._llcrypt registered [ 1312.379737] Key type .llcrypt registered [ 1312.686388] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1312.701510] alg: No test for adler32 (adler32-zlib) [ 1313.693514] LNet: 63259:0:(config.c:1608:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1313.712815] LNet: Added LNI 192.168.201.16@tcp [8/256/0/180] [ 1313.714895] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1313.764830] LNet: 63273:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1313.770462] LNetError: 63273:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1313.777727] LNet: Removed LNI 192.168.201.16@tcp [ 1313.786963] LNet: Removed LNI 10.1.2.3@tcp [ 1314.761993] Key type .llcrypt unregistered [ 1314.763968] Key type ._llcrypt unregistered [ 1321.917601] Key type ._llcrypt registered [ 1321.920926] Key type .llcrypt registered [ 1322.238495] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1322.262492] alg: No test for adler32 (adler32-zlib) [ 1323.289844] LNet: 63534:0:(config.c:1608:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1323.327125] LNet: Added LNI 192.168.201.16@tcp [8/256/0/180] [ 1323.344898] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1323.435770] LNet: 63548:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1323.446340] LNetError: 63548:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1323.455996] LNet: Removed LNI 192.168.201.16@tcp [ 1323.468571] LNet: Removed LNI 10.1.2.3@tcp [ 1324.420663] Key type .llcrypt unregistered [ 1324.422688] Key type ._llcrypt unregistered [ 1332.287769] Key type ._llcrypt registered [ 1332.289769] Key type .llcrypt registered [ 1332.442965] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1332.463558] alg: No test for adler32 (adler32-zlib) [ 1333.398231] LNet: 63812:0:(config.c:1608:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1333.407368] LNet: Added LNI 192.168.201.16@tcp [8/256/0/180] [ 1333.409385] LNetError: 63812:0:(config.c:1593:lnet_inet_select()) ksocklnd: failed to find UP interface test1pg [ 1333.414247] LNetError: Error -100 starting up LNI tcp [ 1333.426757] LNet: Removed LNI 192.168.201.16@tcp [ 1333.945489] Key type .llcrypt unregistered [ 1333.946642] Key type ._llcrypt unregistered [ 1345.915616] Key type ._llcrypt registered [ 1345.919281] Key type .llcrypt registered [ 1347.183611] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 20:47:08 (1778287628) [ 1348.087522] Key type .llcrypt unregistered [ 1348.095388] Key type ._llcrypt unregistered [ 1353.792922] Key type ._llcrypt registered [ 1353.800601] Key type .llcrypt registered [ 1354.120655] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1354.155636] alg: No test for adler32 (adler32-zlib) [ 1355.136910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1355.177987] LNet: Added LNI 192.168.201.16@tcp [8/256/0/180] [ 1362.975947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.116@tcp [ 1363.604613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.16@tcp -d 192.168.201.116@tcp -m GET -e network_timeout -r 1 [ 1363.631696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.16@tcp -d 192.168.201.16@tcp -m GET -e network_timeout -r 1 [ 1363.672262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.116@tcp [ 1363.689190] LNet: There was an unexpected network error while writing to 192.168.201.116: rc = -22 [ 1363.891849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1363.920914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1363.947685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1363.980374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1364.009655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1364.040619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1364.070355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1364.098035] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1364.115644] LNet: 65069:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1364.137745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1364.165618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1364.203141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1364.231814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1364.262789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1364.304648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1364.624691] LNet: 65108:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1364.637426] LNetError: 65108:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1364.656841] LNet: Removed LNI 192.168.201.16@tcp [ 1365.155637] Key type .llcrypt unregistered [ 1365.162453] Key type ._llcrypt unregistered [ 1385.898521] Key type ._llcrypt registered [ 1385.901458] Key type .llcrypt registered [ 1386.080978] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1386.095214] alg: No test for adler32 (adler32-zlib) [ 1387.102290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1387.135571] LNet: Added LNI 192.168.201.16@tcp [8/256/0/180] [ 1395.724935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.116@tcp [ 1405.563443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1405.649726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1405.693461] LNet: Added LNI 192.168.201.16@tcp1 [8/256/0/180] [ 1407.256472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.16@tcp -d 192.168.201.116@tcp -m GET -e network_timeout -r 1 [ 1407.323470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.16@tcp -d 192.168.201.116@tcp1 -m GET -e network_timeout -r 1 [ 1407.355583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.16@tcp -d 192.168.201.16@tcp -m GET -e network_timeout -r 1 [ 1407.387251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.16@tcp -d 192.168.201.16@tcp1 -m GET -e network_timeout -r 1 [ 1407.427634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.16@tcp1 -d 192.168.201.116@tcp -m GET -e network_timeout -r 1 [ 1407.470539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.16@tcp1 -d 192.168.201.116@tcp1 -m GET -e network_timeout -r 1 [ 1407.512967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.16@tcp1 -d 192.168.201.16@tcp -m GET -e network_timeout -r 1 [ 1407.570774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.16@tcp1 -d 192.168.201.16@tcp1 -m GET -e network_timeout -r 1 [ 1407.664431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.116@tcp [ 1407.705767] LNet: There was an unexpected network error while writing to 192.168.201.116: rc = -22 [ 1407.961647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1407.996600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1408.021802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1408.049993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1408.085582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1408.122627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1408.158985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1408.193865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1408.214339] LNet: 65987:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1408.238857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1408.277569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1408.307389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1408.335367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1408.373598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1408.413217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1413.086256] LNet: There was an unexpected network error while writing to 192.168.201.116: rc = -22 [ 1413.544417] LNet: 66061:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1413.558674] LNetError: 66061:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1413.570049] LNet: Removed LNI 192.168.201.16@tcp [ 1413.587909] LNet: Removed LNI 192.168.201.16@tcp1 [ 1414.082654] Key type .llcrypt unregistered [ 1414.085386] Key type ._llcrypt unregistered [ 1432.094116] Key type ._llcrypt registered [ 1432.102076] Key type .llcrypt registered [ 1433.526503] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 20:48:34 (1778287714) [ 1434.128991] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1434.148881] alg: No test for adler32 (adler32-zlib) [ 1435.118584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1435.204820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1435.251118] LNet: Added LNI 192.168.201.16@tcp [8/256/0/180] [ 1436.191434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1436.222276] LNet: Added LNI 192.168.201.16@tcp1 [8/256/0/180] [ 1437.262447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.16@tcp [ 1437.336856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1437.408325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1437.448269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1437.492436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1437.526874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1437.564784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.201.16@tcp [ 1437.585890] LNetError: 67161:0:(api-ni.c:4350:lnet_ni_set_healthv()) manually adding local NI 192.168.201.16@tcp to recovery [ 1438.241413] LNet: 1 local NIs in recovery (showing 1): 192.168.201.16@tcp [ 1438.244072] LNet: There was an unexpected network error while writing to 192.168.201.16: rc = -22 [ 1440.287554] LNet: There was an unexpected network error while writing to 192.168.201.16: rc = -22 [ 1441.409767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1444.384021] LNet: There was an unexpected network error while writing to 192.168.201.16: rc = -22 [ 1445.096265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1445.153680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1445.249938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1445.280827] LNet: 67268:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1445.288859] LNetError: 67268:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1445.298914] LNet: Removed LNI 192.168.201.16@tcp [ 1445.317309] LNet: Removed LNI 192.168.201.16@tcp1 [ 1445.366169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1445.430448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1445.465677] LNet: Added LNI 192.168.201.16@tcp [8/256/0/180] [ 1446.153888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1446.175859] LNet: Added LNI 192.168.201.16@tcp1 [8/256/0/180] [ 1447.194960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.16@tcp [ 1447.239944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1447.312649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1447.370632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1447.426497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1447.462607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1447.511513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1447.558507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.201.16@tcp [ 1447.593571] LNetError: 67541:0:(api-ni.c:4350:lnet_ni_set_healthv()) manually adding local NI 192.168.201.16@tcp to recovery [ 1448.482571] LNet: There was an unexpected network error while writing to 192.168.201.16: rc = -22 [ 1451.408876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1454.629399] LNet: There was an unexpected network error while writing to 192.168.201.16: rc = -22 [ 1454.663447] LNet: Skipped 1 previous similar message [ 1459.872568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1459.934272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1460.001891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1467.690612] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 20:49:08 (1778287748) [ 1468.196410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1468.216199] LNet: 68258:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1468.222941] LNetError: 68258:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1468.229150] LNetError: 68258:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1468.233971] LNet: Removed LNI 192.168.201.16@tcp [ 1468.242661] LNet: Removed LNI 192.168.201.16@tcp1 [ 1468.275217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1468.337332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1468.385625] LNet: Added LNI 192.168.201.16@tcp [8/256/0/180] [ 1469.049136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1469.064913] LNet: Added LNI 192.168.201.16@tcp1 [8/256/0/180] [ 1469.897808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.16@tcp [ 1469.980329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1470.020662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1470.073830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1470.107248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1470.147863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1470.197482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.201.16@tcp --health 0 [ 1470.308312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1470.370208] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.16@tcp [ 1471.392718] LNet: There was an unexpected network error while writing to 192.168.201.16: rc = -22 [ 1471.403361] LNet: Skipped 1 previous similar message [ 1485.943876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1486.118811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1486.161831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1486.205940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.201.16@tcp --health 500 [ 1486.344368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1493.173246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1493.250114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1493.275238] LNet: 68750:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1493.286831] LNetError: 68750:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1493.294197] LNetError: 68750:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1493.302752] LNet: Removed LNI 192.168.201.16@tcp [ 1493.319370] LNet: Removed LNI 192.168.201.16@tcp1 [ 1493.345923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1493.423193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1493.477453] LNet: Added LNI 192.168.201.16@tcp [8/256/0/180] [ 1494.510541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1494.528955] LNet: Added LNI 192.168.201.16@tcp1 [8/256/0/180] [ 1495.451719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.16@tcp [ 1495.504934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1495.593501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1495.632520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1495.675688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1495.719354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1495.751512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1495.803258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.201.16@tcp --health 0 [ 1496.479713] LNet: There was an unexpected network error while writing to 192.168.201.16: rc = -22 [ 1496.487235] LNet: Skipped 3 previous similar messages [ 1497.188922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1506.915588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1506.947347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1506.985650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1516.602725] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 20:49:57 (1778287797) [ 1517.270924] LNet: 69784:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1517.286607] LNetError: 69784:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1517.289742] LNetError: 69784:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1517.320647] LNet: Removed LNI 192.168.201.16@tcp [ 1517.341771] LNet: Removed LNI 192.168.201.16@tcp1 [ 1518.400920] Key type .llcrypt unregistered [ 1518.409496] Key type ._llcrypt unregistered [ 1526.905834] Key type ._llcrypt registered [ 1526.911971] Key type .llcrypt registered [ 1527.210802] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1527.238532] alg: No test for adler32 (adler32-zlib) [ 1528.252614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1528.304496] LNet: Added LNI 192.168.201.16@tcp [8/256/0/180] [ 1538.719505] Lustre: DEBUG MARKER: Initial discovery [ 1538.874679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.201.116@tcp [ 1541.591273] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1541.664527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1541.759362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.201.116@tcp [ 1541.838198] LNet: There was an unexpected network error while writing to 192.168.201.116: rc = -22 [ 1541.895399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1542.000988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.201.116@tcp -r 1 -m GET -l 3 [ 1542.033390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.201.116@tcp -r 1 -m GET -e local_error [ 1542.059423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.201.116@tcp -r 1 -m PUT -l 6 [ 1542.694116] LNet: 1 local NIs in recovery (showing 1): 192.168.201.16@tcp [ 1543.895712] Lustre: DEBUG MARKER: Force oleg116-client.virtnet to discover 192.168.201.116@tcp (in background) [ 1544.014169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.201.116@tcp [ 1545.881890] Lustre: DEBUG MARKER: Force oleg116-server to discover 192.168.201.16@tcp [ 1547.167289] LNet: There was an unexpected network error while writing to 192.168.201.116: rc = -22 [ 1548.849786] Lustre: DEBUG MARKER: Wait for 70539 [ 1550.648946] Lustre: DEBUG MARKER: Finished wait on 70539 [ 1551.054688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay del -a [ 1551.108472] LNet: There was an unexpected network error while writing to 192.168.201.116: rc = -22 [ 1551.130258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1551.527648] LNet: 70997:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1551.535926] LNetError: 70997:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1551.569499] LNet: Removed LNI 192.168.201.16@tcp [ 1552.177979] Key type .llcrypt unregistered [ 1552.185293] Key type ._llcrypt unregistered [ 1571.875163] Key type ._llcrypt registered [ 1571.876939] Key type .llcrypt registered [ 1573.352843] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 20:50:54 (1778287854) [ 1574.271896] Key type .llcrypt unregistered [ 1574.283909] Key type ._llcrypt unregistered [ 1581.390114] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1581.563446] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1581.568253] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1581.574582] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1584.174333] Key type ._llcrypt registered [ 1584.176938] Key type .llcrypt registered [ 1584.362964] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1584.382620] alg: No test for adler32 (adler32-zlib) [ 1585.283618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1585.326960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1585.345885] LNet: Added LNI 192.168.201.16@tcp [8/256/0/180] [ 1586.217858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1586.241937] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1595.427481] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 20:51:16 (1778287876) [ 1595.810678] LNet: 73071:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1595.814437] LNetError: 73071:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1595.824501] LNet: Removed LNI 192.168.201.16@tcp [ 1595.849182] LNet: Removed LNI 10.1.2.3@tcp [ 1596.557429] Key type .llcrypt unregistered [ 1596.562367] Key type ._llcrypt unregistered [ 1604.937766] Key type ._llcrypt registered [ 1604.942625] Key type .llcrypt registered [ 1605.256571] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1605.284404] alg: No test for adler32 (adler32-zlib) [ 1606.341637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1606.434924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1606.491433] LNet: Added LNI 192.168.201.16@tcp [8/256/0/180] [ 1607.465514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1607.484867] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1608.874302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.201.16@tcp [ 1609.951157] LNetError: Unexpected error -22 connecting to 192.168.201.16@tcp at host 192.168.201.16:988 [ 1615.796392] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 20:51:37 (1778287897) [ 1616.129670] LNet: 74205:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1616.137942] LNetError: 74205:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1616.145898] LNet: Removed LNI 192.168.201.16@tcp [ 1616.180963] LNet: Removed LNI 10.1.2.3@tcp [ 1616.857470] Key type .llcrypt unregistered [ 1616.862052] Key type ._llcrypt unregistered [ 1623.528610] Key type ._llcrypt registered [ 1623.531578] Key type .llcrypt registered [ 1623.883853] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1623.930268] alg: No test for adler32 (adler32-zlib) [ 1624.881893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1624.947572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1624.977913] LNet: Added LNI 192.168.201.16@tcp1 [8/256/0/180] [ 1625.883293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1625.920483] LNet: Added LNI 192.168.201.16@tcp2 [8/256/0/180] [ 1626.868222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.201.16@tcp1 --nid 192.168.201.16@tcp2 [ 1631.651942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.16@tcp1 192.168.201.16@tcp2 [ 1631.732427] LNetError: 75040:0:(lib-move.c:2266:lnet_handle_find_routed_path()) Unable to send message from 192.168.201.16@tcp1 to 192.168.201.16@tcp2 - Route table may be misconfigured [ 1631.782650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.16@tcp2 192.168.201.16@tcp1 [ 1639.409333] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 20:52:00 (1778287920) [ 1639.535928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1639.579743] LNet: 75610:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1639.583551] LNetError: 75610:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1639.594281] LNet: Removed LNI 192.168.201.16@tcp1 [ 1640.613443] LNet: Removed LNI 192.168.201.16@tcp2 [ 1640.644901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1640.740503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1640.793909] LNet: Added LNI 192.168.201.16@tcp [8/256/0/180] [ 1641.556812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1641.578905] LNet: Added LNI 192.168.201.16@tcp1 [8/256/0/180] [ 1642.344809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.16@tcp [ 1642.376150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.16@tcp 192.168.201.16@tcp [ 1642.413614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.16@tcp1 192.168.201.16@tcp1 [ 1642.451202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.201.16@tcp -d 192.168.201.16@tcp -e network_timeout [ 1642.476657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.201.16@tcp -d 192.168.201.16@tcp1 -e network_timeout [ 1642.501762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.201.16@tcp1 -d 192.168.201.16@tcp -e network_timeout [ 1642.531646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.201.16@tcp1 -d 192.168.201.16@tcp1 -e network_timeout [ 1642.563162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.16@tcp [ 1642.578580] LNet: There was an unexpected network error while writing to 192.168.201.16: rc = -22 [ 1642.598286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1642.682359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1642.749626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1642.788853] LNet: 1 local NIs in recovery (showing 1): 192.168.201.16@tcp [ 1648.954366] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 20:52:10 (1778287930) [ 1649.123167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1649.172173] LNet: 76475:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1649.191190] LNetError: 76475:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1649.196600] LNetError: 76475:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1649.206565] LNet: Removed LNI 192.168.201.16@tcp [ 1650.279068] LNet: Removed LNI 192.168.201.16@tcp1 [ 1650.308597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1650.472277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1650.805270] LNet: 76513:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1651.248815] Key type .llcrypt unregistered [ 1651.256024] Key type ._llcrypt unregistered [ 1664.478884] Key type ._llcrypt registered [ 1664.480941] Key type .llcrypt registered [ 1666.074431] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 20:52:26 (1778287946) [ 1666.819386] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1666.864932] alg: No test for adler32 (adler32-zlib) [ 1667.988950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1669.591176] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1672.479412] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 20:52:32 (1778287952) [ 1672.564343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1672.626797] LNet: 77604:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1672.680939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1672.801920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1672.926220] LNet: Added LNI 192.168.201.16@tcp [8/256/0/180] [ 1674.194108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1674.231144] LNet: Added LNI 192.168.201.16@tcp1 [8/256/0/180] [ 1675.687518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.16@tcp [ 1675.770638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.16@tcp1 [ 1675.808234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.16@tcp1 [ 1683.729444] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 20:52:44 (1778287964) [ 1685.813823] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg116-server [ 1688.266865] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 20:52:48 (1778287968) [ 1690.450686] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg116-server [ 1692.904074] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 20:52:53 (1778287973) [ 1696.363509] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg116-server [ 1698.551273] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 20:52:59 (1778287979) [ 1700.686891] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg116-server [ 1702.277331] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 20:53:03 (1778287983) [ 1704.038671] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg116-server [ 1705.935796] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 20:53:07 (1778287987) [ 1707.503759] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg116-server [ 1709.295507] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 20:53:10 (1778287990) [ 1712.103764] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg116-server [ 1714.101768] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 20:53:15 (1778287995) [ 1715.970378] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg116-server [ 1717.889162] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 20:53:18 (1778287998) [ 1719.702362] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg116-server [ 1721.561799] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 20:53:22 (1778288002) [ 1721.674108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1721.708993] LNet: 81484:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1721.718026] LNetError: 81484:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1721.727269] LNet: Removed LNI 192.168.201.16@tcp [ 1722.817244] LNet: Removed LNI 192.168.201.16@tcp1 [ 1722.871617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1722.962391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1723.002550] LNet: Added LNI 192.168.201.16@tcp [8/256/0/180] [ 1723.798863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1723.870667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.16@tcp [ 1724.086521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1724.103879] LNet: 81637:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1724.108309] LNetError: 81637:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1724.113734] LNetError: 81637:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1724.123720] LNet: Removed LNI 192.168.201.16@tcp [ 1724.138521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1724.205337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1724.246680] LNet: Added LNI 192.168.201.16@tcp [8/256/0/180] [ 1724.946099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.16@tcp [ 1725.124100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1725.162933] LNetError: 81782:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1725.177923] LNet: Removed LNI 192.168.201.16@tcp [ 1725.195115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1725.274353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1725.299262] LNet: Added LNI 192.168.201.16@tcp [8/256/0/180] [ 1726.059232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1726.177765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.16@tcp [ 1726.501728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1726.548979] LNet: 81929:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1726.554887] LNet: 81929:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1727.591286] LNet: Removed LNI 192.168.201.16@tcp [ 1727.636200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1727.710350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1727.747741] LNet: Added LNI 192.168.201.16@tcp [8/256/0/180] [ 1728.817471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.16@tcp [ 1729.208145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1729.261068] LNetError: 82075:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1729.266900] LNetError: 82075:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1729.308429] LNet: Removed LNI 192.168.201.16@tcp [ 1729.349890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1729.456790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1729.494085] LNet: Added LNI 192.168.201.16@tcp [8/256/0/180] [ 1730.403051] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1730.552802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.16@tcp [ 1730.924928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1730.962117] LNet: 82222:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1730.970487] LNet: 82222:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1732.010080] LNet: Removed LNI 192.168.201.16@tcp [ 1732.051202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1732.160916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1732.230987] LNet: Added LNI 192.168.201.16@tcp [8/256/0/180] [ 1733.311400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.16@tcp [ 1733.661676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1733.712911] LNetError: 82368:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1733.720205] LNetError: 82368:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1733.752278] LNet: Removed LNI 192.168.201.16@tcp [ 1733.769061] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1733.850641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1733.879612] LNet: Added LNI 192.168.201.16@tcp [8/256/0/180] [ 1734.779750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1734.887961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.16@tcp [ 1735.165951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1736.231459] LNet: Removed LNI 192.168.201.16@tcp [ 1736.271465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1736.382077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1736.401526] LNet: Added LNI 192.168.201.16@tcp [8/256/0/180] [ 1737.367841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.16@tcp [ 1737.629720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1738.726493] LNet: Removed LNI 192.168.201.16@tcp [ 1738.774260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1738.840896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1738.870703] LNet: Added LNI 192.168.201.16@tcp [8/256/0/180] [ 1739.635647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1739.695368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.16@tcp [ 1739.982239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1740.002934] LNet: 82808:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1740.009774] LNet: 82808:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 1741.105354] LNet: Removed LNI 192.168.201.16@tcp [ 1741.178260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1741.263491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1741.302257] LNet: Added LNI 192.168.201.16@tcp [8/256/0/180] [ 1742.047618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.16@tcp [ 1742.356904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1742.379866] LNetError: 82954:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1742.388457] LNetError: 82954:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 1743.462260] LNet: Removed LNI 192.168.201.16@tcp [ 1743.525616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1743.620239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1743.647677] LNet: Added LNI 192.168.201.16@tcp [8/256/0/180] [ 1744.213342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1744.306591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.16@tcp [ 1744.625491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1745.704178] LNet: Removed LNI 192.168.201.16@tcp [ 1745.776468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1745.833560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1745.907489] LNet: Added LNI 192.168.201.16@tcp [8/256/0/180] [ 1746.973726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.16@tcp [ 1747.330721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1748.393633] LNet: Removed LNI 192.168.201.16@tcp [ 1748.461598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1748.549811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1748.576856] LNet: Added LNI 192.168.201.16@tcp [8/256/0/180] [ 1749.396291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1749.467826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1749.498172] LNet: Removed LNI 192.168.201.16@tcp [ 1749.525241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1749.594874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1749.640339] LNet: Added LNI 192.168.201.16@tcp [8/256/0/180] [ 1750.291420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1757.413392] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 20:53:58 (1778288038) [ 1757.546286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1757.588373] LNet: 84083:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1757.594229] LNet: 84083:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 1757.611059] LNet: Removed LNI 192.168.201.16@tcp [ 1757.626596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1757.664180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1757.679882] LNet: Added LNI 192.168.201.16@tcp231 [8/256/0/180] [ 1758.974049] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1759.007509] LNetError: 84223:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1759.013430] LNetError: 84223:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 1759.045634] LNet: Removed LNI 192.168.201.16@tcp231 [ 1759.083554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1759.236424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1759.288276] LNet: Added LNI 192.168.201.16@tcp231 [8/256/0/180] [ 1759.295233] LNet: 84231:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1759.755958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1759.802617] LNet: Removed LNI 192.168.201.16@tcp231 [ 1759.824929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1759.860312] LNet: Added LNI 192.168.201.16@tcp231 [8/256/0/180] [ 1761.246314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1761.294294] LNet: Removed LNI 192.168.201.16@tcp231 [ 1761.330587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1761.427519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1761.461359] LNet: Added LNI 192.168.201.16@tcp231 [8/256/0/0] [ 1761.465518] LNet: 84383:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1761.998294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1762.033621] LNet: Removed LNI 192.168.201.16@tcp231 [ 1762.060437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1762.097459] LNet: Added LNI 192.168.201.16@tcp231 [8/256/0/0] [ 1762.935311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1762.956748] LNet: Removed LNI 192.168.201.16@tcp231 [ 1762.981355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1763.052379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1763.072496] LNet: Added LNI 192.168.201.16@tcp231 [8/256/0/60] [ 1763.084577] LNet: 84535:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1763.469355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1763.506373] LNet: Removed LNI 192.168.201.16@tcp231 [ 1763.534418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1763.558611] LNet: Added LNI 192.168.201.16@tcp231 [8/256/0/60] [ 1764.812907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1764.876421] LNet: Removed LNI 192.168.201.16@tcp231 [ 1764.922673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1765.012719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1765.044628] LNet: Added LNI 192.168.201.16@tcp231 [8/256/0/180] [ 1765.666863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1765.692708] LNet: Removed LNI 192.168.201.16@tcp231 [ 1765.710675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1765.731526] LNet: Added LNI 192.168.201.16@tcp231 [8/256/0/180] [ 1773.526990] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 20:54:14 (1778288054) [ 1773.726924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1773.747447] LNet: Removed LNI 192.168.201.16@tcp231 [ 1773.760453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1773.814304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1773.835198] LNet: Added LNI 192.168.201.16@tcp232 [8/256/0/180] [ 1774.516256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1774.558646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1774.594049] LNet: Removed LNI 192.168.201.16@tcp232 [ 1774.610950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1774.643241] LNet: Added LNI 192.168.201.16@tcp232 [8/256/0/180] [ 1775.252372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1775.280431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1775.342501] LNet: Removed LNI 192.168.201.16@tcp232 [ 1775.376291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1775.439371] LNet: Added LNI 192.168.201.16@tcp232 [8/256/0/180] [ 1776.240497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1776.298622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1776.328199] LNet: Removed LNI 192.168.201.16@tcp232 [ 1776.367366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1776.405102] LNet: Added LNI 192.168.201.16@tcp232 [8/256/0/180] [ 1777.190576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1777.210514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1777.235971] LNet: Removed LNI 192.168.201.16@tcp232 [ 1777.247265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1783.268631] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 20:54:24 (1778288064) [ 1784.960743] Key type .llcrypt unregistered [ 1784.964134] Key type ._llcrypt unregistered [ 1793.584755] Key type ._llcrypt registered [ 1793.591222] Key type .llcrypt registered [ 1793.872504] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1793.905916] alg: No test for adler32 (adler32-zlib) [ 1794.890211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1794.942915] LNet: Added LNI 192.168.201.16@tcp [8/256/0/180] [ 1804.079920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.116@tcp [ 1814.118975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1814.177931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1814.222448] LNet: Added LNI 192.168.201.16@tcp1 [8/256/0/180] [ 1815.883856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.16@tcp -d 192.168.201.116@tcp -m GET -e local_interrupt -r 2 [ 1815.921868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.16@tcp -d 192.168.201.116@tcp1 -m GET -e local_interrupt -r 2 [ 1815.959821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.16@tcp -d 192.168.201.16@tcp -m GET -e local_interrupt -r 2 [ 1816.010589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.16@tcp -d 192.168.201.16@tcp1 -m GET -e local_interrupt -r 2 [ 1816.083195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.16@tcp1 -d 192.168.201.116@tcp -m GET -e local_interrupt -r 2 [ 1816.139958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.16@tcp1 -d 192.168.201.116@tcp1 -m GET -e local_interrupt -r 2 [ 1816.192715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.16@tcp1 -d 192.168.201.16@tcp -m GET -e local_interrupt -r 2 [ 1816.243797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.16@tcp1 -d 192.168.201.16@tcp1 -m GET -e local_interrupt -r 2 [ 1816.296839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.16@tcp 192.168.201.116@tcp [ 1816.320706] LNet: There was an unexpected network error while writing to 192.168.201.116: rc = -22 [ 1816.333820] LNet: 1 local NIs in recovery (showing 1): 192.168.201.16@tcp [ 1816.623071] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1816.661759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1816.736957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1816.856512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1816.888938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1816.939303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1817.005704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1817.076452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1817.113384] LNet: 87188:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1817.163548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1817.232888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1817.285863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1817.337895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1817.410699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1817.502194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1818.200832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.16@tcp -d 192.168.201.116@tcp -m GET -e local_dropped -r 2 [ 1818.229245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.16@tcp -d 192.168.201.116@tcp1 -m GET -e local_dropped -r 2 [ 1818.259158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.16@tcp -d 192.168.201.16@tcp -m GET -e local_dropped -r 2 [ 1818.284339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.16@tcp -d 192.168.201.16@tcp1 -m GET -e local_dropped -r 2 [ 1818.308935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.16@tcp1 -d 192.168.201.116@tcp -m GET -e local_dropped -r 2 [ 1818.351888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.16@tcp1 -d 192.168.201.116@tcp1 -m GET -e local_dropped -r 2 [ 1818.388710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.16@tcp1 -d 192.168.201.16@tcp -m GET -e local_dropped -r 2 [ 1818.424852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.16@tcp1 -d 192.168.201.16@tcp1 -m GET -e local_dropped -r 2 [ 1818.461205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.16@tcp 192.168.201.116@tcp [ 1818.485327] LNet: There was an unexpected network error while writing to 192.168.201.116: rc = -22 [ 1818.860925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1818.916853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1818.953233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1819.018275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1819.074335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1819.164880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1819.219446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1819.257884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1819.283651] LNet: 87309:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1819.314194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1819.339373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1819.371895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1819.396642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1819.428759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1819.458771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1820.117752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.16@tcp -d 192.168.201.116@tcp -m GET -e local_aborted -r 2 [ 1820.159070] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.16@tcp -d 192.168.201.116@tcp1 -m GET -e local_aborted -r 2 [ 1820.237323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.16@tcp -d 192.168.201.16@tcp -m GET -e local_aborted -r 2 [ 1820.302970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.16@tcp -d 192.168.201.16@tcp1 -m GET -e local_aborted -r 2 [ 1820.391441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.16@tcp1 -d 192.168.201.116@tcp -m GET -e local_aborted -r 2 [ 1820.463410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.16@tcp1 -d 192.168.201.116@tcp1 -m GET -e local_aborted -r 2 [ 1820.533506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.16@tcp1 -d 192.168.201.16@tcp -m GET -e local_aborted -r 2 [ 1820.594590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.16@tcp1 -d 192.168.201.16@tcp1 -m GET -e local_aborted -r 2 [ 1820.661738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.16@tcp 192.168.201.116@tcp [ 1820.708476] LNet: There was an unexpected network error while writing to 192.168.201.116: rc = -22 [ 1821.104701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1821.126616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1821.160373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1821.186734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1821.232891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1821.255358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1821.275458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1821.297397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1821.314094] LNet: 87429:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1821.332827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1821.378319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1821.419101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1821.446670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1821.533479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1821.572531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1822.320919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.16@tcp -d 192.168.201.116@tcp -m GET -e local_no_route -r 2 [ 1822.353684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.16@tcp -d 192.168.201.116@tcp1 -m GET -e local_no_route -r 2 [ 1822.381696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.16@tcp -d 192.168.201.16@tcp -m GET -e local_no_route -r 2 [ 1822.430972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.16@tcp -d 192.168.201.16@tcp1 -m GET -e local_no_route -r 2 [ 1822.477531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.16@tcp1 -d 192.168.201.116@tcp -m GET -e local_no_route -r 2 [ 1822.522521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.16@tcp1 -d 192.168.201.116@tcp1 -m GET -e local_no_route -r 2 [ 1822.562380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.16@tcp1 -d 192.168.201.16@tcp -m GET -e local_no_route -r 2 [ 1822.617273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.16@tcp1 -d 192.168.201.16@tcp1 -m GET -e local_no_route -r 2 [ 1822.674745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.16@tcp 192.168.201.116@tcp [ 1822.725270] LNet: There was an unexpected network error while writing to 192.168.201.116: rc = -22 [ 1823.055567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1823.094101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1823.141232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1823.191255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1823.255109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1823.278425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1823.311788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1823.380276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1823.424737] LNet: 87549:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1823.468572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1823.520426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1823.572760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1823.615207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1823.661188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1823.727433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1824.577530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.16@tcp -d 192.168.201.116@tcp -m GET -e local_timeout -r 2 [ 1824.638403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.16@tcp -d 192.168.201.116@tcp1 -m GET -e local_timeout -r 2 [ 1824.716428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.16@tcp -d 192.168.201.16@tcp -m GET -e local_timeout -r 2 [ 1824.792536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.16@tcp -d 192.168.201.16@tcp1 -m GET -e local_timeout -r 2 [ 1824.876783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.16@tcp1 -d 192.168.201.116@tcp -m GET -e local_timeout -r 2 [ 1824.935588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.16@tcp1 -d 192.168.201.116@tcp1 -m GET -e local_timeout -r 2 [ 1824.986730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.16@tcp1 -d 192.168.201.16@tcp -m GET -e local_timeout -r 2 [ 1825.073211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.16@tcp1 -d 192.168.201.16@tcp1 -m GET -e local_timeout -r 2 [ 1825.154719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.16@tcp 192.168.201.116@tcp [ 1825.454647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1825.512312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1825.551746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1825.599648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1825.658422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1825.725060] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1825.793507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1825.857843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1825.908495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1825.972576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1826.027972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1826.094776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1826.157747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1826.228646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1827.087456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.16@tcp -d 192.168.201.116@tcp -m GET -e remote_dropped -r 2 [ 1827.171503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.16@tcp -d 192.168.201.116@tcp1 -m GET -e remote_dropped -r 2 [ 1827.266992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.16@tcp -d 192.168.201.16@tcp -m GET -e remote_dropped -r 2 [ 1827.324921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.16@tcp -d 192.168.201.16@tcp1 -m GET -e remote_dropped -r 2 [ 1827.401393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.16@tcp1 -d 192.168.201.116@tcp -m GET -e remote_dropped -r 2 [ 1827.508560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.16@tcp1 -d 192.168.201.116@tcp1 -m GET -e remote_dropped -r 2 [ 1827.575968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.16@tcp1 -d 192.168.201.16@tcp -m GET -e remote_dropped -r 2 [ 1827.652192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.16@tcp1 -d 192.168.201.16@tcp1 -m GET -e remote_dropped -r 2 [ 1827.733472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.16@tcp 192.168.201.116@tcp [ 1827.781657] LNet: There was an unexpected network error while writing to 192.168.201.116: rc = -22 [ 1827.793383] LNet: Skipped 1 previous similar message [ 1827.799953] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.116@tcp [ 1828.269381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1828.341432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1828.382696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1828.429601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1828.470505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1828.523202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1828.571422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1828.659175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1828.698241] LNet: 87789:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1828.713562] LNet: 87789:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1828.753918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1828.816845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1828.869059] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1828.908970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1828.986089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1829.036484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1836.221534] LNet: 87865:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1836.243627] LNetError: 87865:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1836.260618] LNet: Removed LNI 192.168.201.16@tcp [ 1836.274966] LNet: Removed LNI 192.168.201.16@tcp1 [ 1836.897889] Key type .llcrypt unregistered [ 1836.899702] Key type ._llcrypt unregistered [ 1858.213442] Key type ._llcrypt registered [ 1858.214976] Key type .llcrypt registered [ 1860.254411] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 20:55:40 (1778288140) [ 1860.872699] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1860.888753] alg: No test for adler32 (adler32-zlib) [ 1861.849318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1861.918891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1861.941148] LNet: Added LNI 192.168.201.16@tcp [8/256/0/180] [ 1863.126865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1863.174551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.201.16@tcp [ 1863.192766] LNetError: 88844:0:(api-ni.c:4350:lnet_ni_set_healthv()) manually adding local NI 192.168.201.16@tcp to recovery [ 1863.283623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.201.16@tcp [ 1863.365637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1870.794733] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 20:55:51 (1778288151) [ 1870.932484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1870.968122] LNet: 89427:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1870.977529] LNetError: 89427:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1871.000747] LNet: Removed LNI 192.168.201.16@tcp [ 1871.023283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1871.084352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1871.103551] LNet: Added LNI 192.168.201.16@tcp [8/256/0/180] [ 1871.915771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1871.957254] LNet: Added LNI 192.168.201.16@tcp1 [8/256/0/180] [ 1873.133197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.201.16@tcp [ 1873.151745] LNetError: 89700:0:(api-ni.c:4350:lnet_ni_set_healthv()) manually adding local NI 192.168.201.16@tcp to recovery [ 1874.143416] LNet: 1 local NIs in recovery (showing 1): 192.168.201.16@tcp [ 1874.149750] LNet: There was an unexpected network error while writing to 192.168.201.16: rc = -22 [ 1874.464630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.201.16@tcp [ 1883.467199] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 20:56:04 (1778288164) [ 1885.350241] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg116-server [ 1887.282823] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 20:56:08 (1778288168) [ 1888.808779] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg116-server [ 1890.754689] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 20:56:11 (1778288171) [ 1892.390792] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg116-server [ 1894.508388] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 20:56:15 (1778288175) [ 1894.613209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1894.678600] LNet: 91324:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1894.695522] LNetError: 91324:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1894.707582] LNet: Removed LNI 192.168.201.16@tcp [ 1894.737963] LNet: Removed LNI 192.168.201.16@tcp1 [ 1894.780752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1894.872683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1894.899444] LNet: Added LNI 192.168.201.16@tcp [8/256/0/180] [ 1895.666525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1895.750414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1895.805518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1895.884855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1895.940242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1895.976839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1901.856111] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 20:56:23 (1778288183) [ 1901.980959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1902.022265] LNet: 92051:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1902.028665] LNetError: 92051:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1902.036156] LNetError: 92051:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1902.053448] LNet: Removed LNI 192.168.201.16@tcp [ 1902.090211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1902.148073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1902.181121] LNet: Added LNI 192.168.201.16@tcp [8/256/0/180] [ 1908.122843] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 20:56:29 (1778288189) [ 1908.186627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1908.216083] LNet: 92752:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1908.223584] LNetError: 92752:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1908.244921] LNet: Removed LNI 192.168.201.16@tcp [ 1908.276986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1908.391441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1908.417908] LNet: Added LNI 192.168.201.16@tcp [8/256/0/180] [ 1915.550870] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 20:56:36 (1778288196) [ 1917.215625] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 1918.883351] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 20:56:40 (1778288200) [ 1919.258059] LNet: 93794:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1919.263510] LNetError: 93794:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1919.274941] LNet: Removed LNI 192.168.201.16@tcp [ 1919.817784] Key type .llcrypt unregistered [ 1919.821491] Key type ._llcrypt unregistered [ 1926.324874] Key type ._llcrypt registered [ 1926.327887] Key type .llcrypt registered [ 1926.695508] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1926.718900] alg: No test for adler32 (adler32-zlib) [ 1927.758154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1927.792093] LNet: Added LNI 192.168.201.16@tcp [8/256/0/180] [ 1935.220443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.116@tcp [ 1940.704238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.201.116@tcp [ 1941.070165] LNet: 94211:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1941.078159] LNetError: 94211:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1941.106592] LNet: Removed LNI 192.168.201.16@tcp [ 1941.643642] Key type .llcrypt unregistered [ 1941.652602] Key type ._llcrypt unregistered [ 1953.078422] Key type ._llcrypt registered [ 1953.081404] Key type .llcrypt registered [ 1954.402836] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 1956.217472] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 1958.373357] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 20:57:19 (1778288239) [ 1958.844772] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1958.872393] alg: No test for adler32 (adler32-zlib) [ 1959.933382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1960.116683] LNet: 95300:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1960.917515] Key type .llcrypt unregistered [ 1960.921125] Key type ._llcrypt unregistered [ 1968.223098] Key type ._llcrypt registered [ 1968.225943] Key type .llcrypt registered [ 1968.507926] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1968.534880] alg: No test for adler32 (adler32-zlib) [ 1969.564315] LNet: Added LNI 192.168.201.16@tcp [8/256/0/180] [ 1969.570515] LNet: 95556:0:(config.c:1245:lnet_parse_routes()) Kernel parsing of LNet routes is deprecated. Consider converting to LNet YAML configuration. [ 1969.583294] LNet: 95556:0:(router.c:728:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 1969.639825] LNetError: 95568:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.201.18@tcp is being used as a gateway but routing feature is not turned on [ 1969.655597] LNetError: 95568:0:(lib-lnet.h:1336:lnet_set_route_aliveness()) route to o2ib through 192.168.201.18@tcp has gone from up to down [ 1970.205330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1970.223445] LNet: 95614:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1970.227602] LNetError: 95614:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1970.260132] LNet: Removed LNI 192.168.201.16@tcp [ 1970.329066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1970.440190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1970.508691] LNet: Added LNI 192.168.201.16@tcp [8/256/0/180] [ 1970.525980] LNet: 95621:0:(router.c:728:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 1970.540668] LNet: 95621:0:(router.c:728:lnet_add_route()) Skipped 4 previous similar messages [ 1970.555108] LNetError: 95621:0:(peer.c:366:lnet_peer_ni_del_locked()) Peer NI 192.168.201.17@tcp is a gateway. Can not delete it [ 1970.565896] LNetError: 95621:0:(peer.c:2166:lnet_del_peer_ni()) 192.168.201.17@tcp is a router. Can not be deleted [ 1970.582868] LNet: 95621:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1970.605960] LNetError: 95618:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.201.21@tcp is being used as a gateway but routing feature is not turned on [ 1970.622741] LNetError: 95618:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 7 previous similar messages [ 1970.632197] LNetError: 95618:0:(lib-lnet.h:1336:lnet_set_route_aliveness()) route to o2ib through 192.168.201.21@tcp has gone from up to down [ 1970.651638] LNetError: 95618:0:(lib-lnet.h:1336:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 1977.740845] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 20:57:38 (1778288258) [ 1979.845750] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg116-server [ 1981.267669] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 20:57:42 (1778288262) [ 1983.098301] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg116-server [ 1984.655472] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 20:57:46 (1778288266) [ 1986.946503] Lustre: Lustre: Build Version: 2.17.52_6_gcfb9284 [ 1988.791228] Key type lgssc registered [ 1990.095627] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2009.603392] Key type lgssc unregistered [ 2010.004750] LNet: 98041:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2010.045802] LNetError: 98041:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2011.110626] LNet: Removed LNI 192.168.201.16@tcp [ 2011.887219] Key type .llcrypt unregistered [ 2011.893955] Key type ._llcrypt unregistered [ 2025.262209] Key type ._llcrypt registered [ 2025.268746] Key type .llcrypt registered [ 2026.779913] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 20:58:27 (1778288307) [ 2027.264481] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2027.275833] alg: No test for adler32 (adler32-zlib) [ 2028.227655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2028.302468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2028.370904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2028.439413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2028.493146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2035.279150] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 20:58:36 (1778288316) [ 2035.485090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2035.553238] LNet: 99446:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2035.577597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2035.637489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2035.661193] LNet: Added LNI 192.168.201.16@tcp [8/256/0/180] [ 2036.710600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2036.724926] LNet: 99590:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2036.729169] LNetError: 99590:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2036.738535] LNet: Removed LNI 192.168.201.16@tcp [ 2036.752793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2036.825450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 2036.852890] LNet: Added LNI 192.168.201.16@tcp [8/256/0/180] [ 2037.355552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2037.394666] LNetError: 99618:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2037.410515] LNet: Removed LNI 192.168.201.16@tcp [ 2037.432809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2037.511648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 2037.545715] LNet: Added LNI 192.168.201.16@tcp [8/256/0/180] [ 2047.092648] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 20:58:47 (1778288327) [ 2047.663591] LNet: 100211:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2047.677684] LNet: 100211:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 2047.690388] LNetError: 100211:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2047.744326] LNet: Removed LNI 192.168.201.16@tcp [ 2049.346774] Key type .llcrypt unregistered [ 2049.349780] Key type ._llcrypt unregistered [ 2059.262125] Key type ._llcrypt registered [ 2059.268354] Key type .llcrypt registered [ 2059.547211] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2059.579173] alg: No test for adler32 (adler32-zlib) [ 2060.556125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2060.655991] LNet: Added LNI 192.168.201.16@tcp [8/256/0/180] [ 2060.664871] LNet: *** cfs_fail_loc=e003, val=100*** [ 2061.605987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.16@tcp [ 2061.629262] LNet: *** cfs_fail_loc=e003, val=100*** [ 2061.634515] LNet: Skipped 2 previous similar messages [ 2061.649287] Ping reply received with mapping: [ 2061.652445] found metadata reply NI[192.168.201.16@tcp] [ 2061.655800] 10 c9 a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 2061.661531] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 2061.668868] 5a 5a 5a 5a 5a 5a 5a 5a [ 2061.913179] LNet: 100609:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2061.919133] LNetError: 100609:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2061.939977] LNet: Removed LNI 192.168.201.16@tcp [ 2062.658524] Key type .llcrypt unregistered [ 2062.665197] Key type ._llcrypt unregistered [ 2075.182641] Key type ._llcrypt registered [ 2075.183993] Key type .llcrypt registered [ 2076.359318] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 20:59:17 (1778288357) [ 2076.983481] Key type .llcrypt unregistered [ 2076.985927] Key type ._llcrypt unregistered [ 2084.274966] Key type ._llcrypt registered [ 2084.280345] Key type .llcrypt registered [ 2084.605182] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2084.643459] alg: No test for adler32 (adler32-zlib) [ 2085.938174] LNetError: 101655:0:(api-ni.c:2731:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 2086.715471] Key type .llcrypt unregistered [ 2086.716938] Key type ._llcrypt unregistered [ 2093.330709] Key type ._llcrypt registered [ 2093.333505] Key type .llcrypt registered [ 2094.952591] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 20:59:35 (1778288375) [ 2095.821366] Key type .llcrypt unregistered [ 2095.829948] Key type ._llcrypt unregistered [ 2103.670657] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2103.685380] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2106.533407] Key type ._llcrypt registered [ 2106.549703] Key type .llcrypt registered [ 2106.823504] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2106.860792] alg: No test for adler32 (adler32-zlib) [ 2107.937054] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2108.105180] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2119.304370] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 20:59:59 (1778288399) [ 2119.440973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2119.497335] LNet: 103429:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2119.509518] LNetError: 103429:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2119.536293] LNet: Removed LNI 10.1.2.3@tcp [ 2119.567502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2120.176480] Key type .llcrypt unregistered [ 2120.178755] Key type ._llcrypt unregistered [ 2126.701766] Key type ._llcrypt registered [ 2126.706321] Key type .llcrypt registered [ 2126.889439] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2126.900897] alg: No test for adler32 (adler32-zlib) [ 2127.865390] LNet: Added LNI 192.168.201.16@tcp [8/256/0/180] [ 2127.926520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.16@tcp [ 2128.114291] LNet: 103724:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2128.122351] LNetError: 103724:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2129.189958] LNet: Removed LNI 192.168.201.16@tcp [ 2130.096716] Key type .llcrypt unregistered [ 2130.099079] Key type ._llcrypt unregistered [ 2142.870240] Key type ._llcrypt registered [ 2142.872503] Key type .llcrypt registered [ 2144.276462] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 21:00:25 (1778288425) [ 2144.851375] Key type .llcrypt unregistered [ 2144.853634] Key type ._llcrypt unregistered [ 2151.580470] Key type ._llcrypt registered [ 2151.583791] Key type .llcrypt registered [ 2151.860454] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2151.873392] alg: No test for adler32 (adler32-zlib) [ 2159.750415] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 21:00:41 (1778288441) [ 2159.896765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2159.941133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2159.995737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2160.034115] LNet: Added LNI 192.168.201.16@tcp [8/256/0/180] [ 2160.979328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2161.043465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 2161.098835] LNet: Removed LNI 192.168.201.16@tcp [ 2161.448647] LNet: 105467:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2161.899520] Key type .llcrypt unregistered [ 2161.901519] Key type ._llcrypt unregistered [ 2175.080127] Key type ._llcrypt registered [ 2175.083139] Key type .llcrypt registered [ 2176.631238] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 21:00:57 (1778288457) [ 2177.122382] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2177.140171] alg: No test for adler32 (adler32-zlib) [ 2178.125490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2178.242509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2178.278936] LNet: Added LNI 192.168.201.16@tcp [8/256/0/180] [ 2179.528374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.16@tcp [ 2179.585045] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.201.16@tcp [ 2179.602985] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 2179.610150] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 2179.618434] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2179, 0, 0 [ 2179.625564] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 2179.630514] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 2179.646203] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 2179.655557] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2179, 0, 0 [ 2179.680130] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 2179.684636] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2179.705626] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 2179.719591] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2179, 0, 2229 [ 2179.733396] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 2179.740900] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2179.756670] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 2179.764494] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2179, 0, 2229 [ 2179.771312] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 2186.550811] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 21:01:07 (1778288467) [ 2187.097675] LNet: 107020:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2187.116990] LNetError: 107020:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2187.137546] LNet: Removed LNI 192.168.201.16@tcp [ 2187.727362] Key type .llcrypt unregistered [ 2187.731150] Key type ._llcrypt unregistered [ 2194.922114] Key type ._llcrypt registered [ 2194.939756] Key type .llcrypt registered [ 2195.170206] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2195.184567] alg: No test for adler32 (adler32-zlib) [ 2196.133838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2196.165678] LNet: Added LNI 192.168.201.16@tcp [8/256/0/180] [ 2203.039643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.116@tcp [ 2211.884748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2211.937799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2211.958858] LNet: Added LNI 192.168.201.16@tcp1 [8/256/0/180] [ 2212.941833] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2213.059100] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2213.077062] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2213.081187] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2215.490664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 2215.520264] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 2226.095270] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 21:01:47 (1778288507) [ 2226.316758] LNet: 108518:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2226.322523] LNetError: 108518:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2226.340895] LNet: Removed LNI 192.168.201.16@tcp [ 2227.360058] LNet: Removed LNI 192.168.201.16@tcp1 [ 2227.370591] LNetError: 108518:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2227.385019] LNetError: 108518:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2227.416342] LNet: Removed LNI 10.1.2.3@tcp99 [ 2228.057888] Key type .llcrypt unregistered [ 2228.062933] Key type ._llcrypt unregistered [ 2234.036473] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2234.044228] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2236.686671] Key type ._llcrypt registered [ 2236.688949] Key type .llcrypt registered [ 2236.927397] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2236.955540] alg: No test for adler32 (adler32-zlib) [ 2237.840878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2237.885312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2237.909358] LNet: Added LNI 192.168.201.16@tcp [8/256/0/180] [ 2238.572675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2238.590892] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2240.064566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 2240.100724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2240.135469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 2240.150830] LNetError: 109266:0:(peer.c:2174:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 2240.189726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2240.236513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.16@tcp [ 2240.870944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2240.954413] LNet: 109286:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2240.958905] LNetError: 109286:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2242.016200] LNet: Removed LNI 192.168.201.16@tcp [ 2242.025614] LNetError: 109286:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2242.055427] LNet: Removed LNI 10.1.2.3@tcp [ 2242.780667] Key type .llcrypt unregistered [ 2242.783458] Key type ._llcrypt unregistered [ 2254.119244] Key type ._llcrypt registered [ 2254.121552] Key type .llcrypt registered [ 2255.462965] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 21:02:16 (1778288536) [ 2255.812574] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2255.825919] alg: No test for adler32 (adler32-zlib) [ 2256.784891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2256.851353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2256.876440] LNet: Added LNI 192.168.201.16@tcp [8/256/0/180] [ 2257.512788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.16@tcp [ 2257.573468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg116-client@tcp [ 2264.000991] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 21:02:25 (1778288545) [ 2264.299259] LNet: 110820:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2264.313450] LNetError: 110820:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2264.369337] LNet: Removed LNI 192.168.201.16@tcp [ 2265.027871] Key type .llcrypt unregistered [ 2265.033334] Key type ._llcrypt unregistered [ 2271.781174] Key type ._llcrypt registered [ 2271.783598] Key type .llcrypt registered [ 2272.016855] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2272.046663] alg: No test for adler32 (adler32-zlib) [ 2272.991636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 2273.027598] LNet: Added LNI 192.168.201.16@tcp [8/256/0/180] [ 2273.060355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 2279.230774] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 21:02:40 (1778288560) [ 2280.604678] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 2282.095401] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 21:02:43 (1778288563) [ 2283.727245] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 2285.662451] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 21:02:46 (1778288566) [ 2285.769695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2285.814598] LNet: 112234:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2285.820886] LNetError: 112234:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2285.839327] LNet: Removed LNI 192.168.201.16@tcp [ 2285.902372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2286.008344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2286.025428] LNet: Added LNI 192.168.201.16@tcp [8/256/0/180] [ 2286.754815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2286.834939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2293.933616] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 21:02:55 (1778288575) [ 2294.165489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2294.182976] LNet: 112951:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2294.192036] LNetError: 112951:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2294.211108] LNet: Removed LNI 192.168.201.16@tcp [ 2294.228252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2294.279732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 2294.321362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2294.982543] Key type .llcrypt unregistered [ 2294.984851] Key type ._llcrypt unregistered [ 2308.357189] Key type ._llcrypt registered [ 2308.361042] Key type .llcrypt registered [ 2309.683827] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 21:03:10 (1778288590) [ 2310.097545] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2310.108976] alg: No test for adler32 (adler32-zlib) [ 2311.058749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2311.110765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2311.123299] LNet: Added LNI 192.168.201.16@tcp [8/256/0/180] [ 2311.906788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2311.948954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.16@tcp [ 2318.685473] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 21:03:19 (1778288599) [ 2318.766788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2318.784287] LNet: 114501:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2318.789948] LNetError: 114501:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2319.848228] LNet: Removed LNI 192.168.201.16@tcp [ 2319.894634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2319.984275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 2320.027312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 2325.985496] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 21:03:27 (1778288607) [ 2326.177289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2326.203635] LNet: 115087:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2326.233777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2326.304338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2326.700338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2326.750156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2326.824673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 2327.281940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2327.315055] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2327.391483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2327.412839] LNet: 115120:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2327.432723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2327.497611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2327.949458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2327.980582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2328.039887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 2328.558464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2328.609330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2335.473994] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 21:03:36 (1778288616) [ 2335.656577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2335.681206] LNet: 115720:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2335.714746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2335.741713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2335.781908] LNet: Added LNI 192.168.201.16@tcp [8/256/0/180] [ 2336.433851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2336.457082] LNet: Added LNI 192.168.201.16@tcp2 [8/256/0/180] [ 2337.224892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2337.268350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.16@tcp [ 2337.461653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2337.479243] LNetError: 116000:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2337.485930] LNet: Removed LNI 192.168.201.16@tcp [ 2337.498748] LNet: Removed LNI 192.168.201.16@tcp2 [ 2337.514679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2337.575672] LNet: Added LNI 192.168.201.16@tcp [8/256/0/180] [ 2337.589445] LNet: Added LNI 192.168.201.16@tcp2 [8/256/0/180] [ 2337.608618] LNet: 116006:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2338.235507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.16@tcp [ 2338.451962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2338.484136] LNetError: 116043:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2338.491951] LNetError: 116043:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2338.498913] LNet: Removed LNI 192.168.201.16@tcp [ 2338.510072] LNet: Removed LNI 192.168.201.16@tcp2 [ 2338.535042] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2338.584896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2338.597637] LNet: Added LNI 192.168.201.16@tcp [8/256/0/180] [ 2339.206670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2339.220808] LNet: Added LNI 192.168.201.16@tcp2 [8/256/0/180] [ 2340.318085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.16@tcp [ 2340.365287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2340.450393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.16@tcp [ 2340.703404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2340.725393] LNet: 116327:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2340.730168] LNet: 116327:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 2340.736059] LNetError: 116327:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2340.745107] LNetError: 116327:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2341.791115] LNet: Removed LNI 192.168.201.16@tcp [ 2342.826797] LNet: Removed LNI 192.168.201.16@tcp2 [ 2342.881436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2342.992117] LNet: Added LNI 192.168.201.16@tcp [8/256/0/180] [ 2343.011096] LNet: Added LNI 192.168.201.16@tcp2 [8/256/0/180] [ 2343.039982] LNet: 116334:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2343.374875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.16@tcp [ 2343.559669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2343.585886] LNet: Removed LNI 192.168.201.16@tcp [ 2344.627456] LNet: Removed LNI 192.168.201.16@tcp2 [ 2344.661792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2344.725699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2344.747225] LNet: Added LNI 192.168.201.16@tcp [8/256/0/180] [ 2345.463368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2345.488718] LNet: Added LNI 192.168.201.16@tcp2 [8/256/0/180] [ 2346.420446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2346.505185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.16@tcp [ 2346.749754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2346.772851] LNetError: 116650:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2346.777892] LNetError: 116650:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 2347.807252] LNet: Removed LNI 192.168.201.16@tcp [ 2347.830390] LNet: Removed LNI 192.168.201.16@tcp2 [ 2347.872235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2348.031993] LNet: Added LNI 192.168.201.16@tcp [8/256/0/180] [ 2348.040932] LNet: Added LNI 192.168.201.16@tcp2 [8/256/0/180] [ 2348.046228] LNet: 116656:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2348.610434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.16@tcp [ 2348.808806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2348.826892] LNet: 116693:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2348.832417] LNet: 116693:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 2349.855321] LNet: Removed LNI 192.168.201.16@tcp [ 2349.872879] LNet: Removed LNI 192.168.201.16@tcp2 [ 2349.910086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2349.971140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2349.998817] LNet: Added LNI 192.168.201.16@tcp [8/256/0/180] [ 2350.660191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2350.688060] LNet: Added LNI 192.168.201.16@tcp2 [8/256/0/180] [ 2351.743970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.16@tcp [ 2351.808757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2351.902249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.16@tcp [ 2352.262566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2353.311236] LNet: Removed LNI 192.168.201.16@tcp [ 2354.341348] LNet: Removed LNI 192.168.201.16@tcp2 [ 2354.403806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2354.508324] LNet: Added LNI 192.168.201.16@tcp [8/256/0/180] [ 2354.523337] LNet: Added LNI 192.168.201.16@tcp2 [8/256/0/180] [ 2354.543373] LNet: 116984:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2355.267900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.16@tcp [ 2355.622732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2355.661461] LNetError: 117021:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2355.671481] LNetError: 117021:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 2355.681768] LNet: Removed LNI 192.168.201.16@tcp [ 2356.711394] LNet: Removed LNI 192.168.201.16@tcp2 [ 2356.752736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2356.822273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2356.842126] LNet: Added LNI 192.168.201.16@tcp [8/256/0/180] [ 2357.729962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2357.766781] LNet: Added LNI 192.168.201.16@tcp2 [8/256/0/180] [ 2358.776243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2358.803574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2358.876817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.16@tcp [ 2359.300957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2360.351198] LNet: Removed LNI 192.168.201.16@tcp [ 2360.367390] LNet: Removed LNI 192.168.201.16@tcp2 [ 2360.451917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2360.545850] LNet: Added LNI 192.168.201.16@tcp [8/256/0/180] [ 2360.556632] LNet: Added LNI 192.168.201.16@tcp2 [8/256/0/180] [ 2360.562899] LNet: 117324:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2361.044959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.16@tcp [ 2361.344375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2362.399410] LNet: Removed LNI 192.168.201.16@tcp [ 2362.420760] LNet: Removed LNI 192.168.201.16@tcp2 [ 2362.468580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2362.529248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2362.551617] LNet: Added LNI 192.168.201.16@tcp [8/256/0/180] [ 2363.279564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2363.296493] LNet: Added LNI 192.168.201.16@tcp2 [8/256/0/180] [ 2364.070958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.16@tcp [ 2364.103609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2364.153650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2364.206665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.16@tcp [ 2364.483731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2365.540738] LNet: Removed LNI 192.168.201.16@tcp [ 2366.566398] LNet: Removed LNI 192.168.201.16@tcp2 [ 2366.659539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2366.802825] LNet: Added LNI 192.168.201.16@tcp [8/256/0/180] [ 2366.816640] LNet: Added LNI 192.168.201.16@tcp2 [8/256/0/180] [ 2367.384728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.16@tcp [ 2374.254104] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 2376.670138] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 21:04:17 (1778288657) [ 2377.196807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2377.222682] LNet: 118442:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2377.234758] LNet: 118442:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 5 previous similar messages [ 2377.245294] LNetError: 118442:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2377.255570] LNetError: 118442:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 2377.264659] LNet: Removed LNI 192.168.201.16@tcp [ 2378.281541] LNet: Removed LNI 192.168.201.16@tcp2 [ 2378.328276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2378.820262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2378.871624] LNet: Added LNI 192.168.201.16@tcp [8/256/0/180] [ 2379.717549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2379.740568] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2380.665388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2380.684640] LNet: Removed LNI 192.168.201.16@tcp [ 2380.708531] LNet: Removed LNI 10.1.2.3@tcp [ 2380.737702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2380.795419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 2380.829324] LNet: Added LNI 192.168.201.16@tcp [8/256/0/180] [ 2380.840894] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2381.277981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 2381.297648] LNet: Removed LNI 192.168.201.16@tcp [ 2381.315475] LNet: Removed LNI 10.1.2.3@tcp [ 2387.972251] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 21:04:29 (1778288669) [ 2388.211763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2388.268098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2388.359659] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2388.504126] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2388.510632] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2388.517591] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2390.924407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2390.962621] LNet: Added LNI 192.168.201.16@tcp [8/256/0/180] [ 2391.953256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2391.974419] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2393.954463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.16@tcp [ 2393.979970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 2394.043182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.201.16@tcp -m PUT -e local_timeout -r 1 [ 2394.101835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 2394.168946] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2394.172739] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2394.394738] LNetError: 119383:0:(lib-move.c:4969:lnet_parse()) 192.168.201.16@tcp, src 192.168.201.16@tcp: Dropping PUT (error -108 looking up sender) [ 2396.447134] LNet: Removed LNI 192.168.201.16@tcp [ 2396.456912] LNet: Removed LNI 10.1.2.3@tcp [ 2396.814604] Key type .llcrypt unregistered [ 2396.818097] Key type ._llcrypt unregistered [ 2407.538881] Key type ._llcrypt registered [ 2407.541323] Key type .llcrypt registered [ 2409.101125] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 21:04:50 (1778288690) [ 2409.604583] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2409.636384] alg: No test for adler32 (adler32-zlib) [ 2410.537436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2410.917706] LNet: 120693:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2411.646388] Key type .llcrypt unregistered [ 2411.651279] Key type ._llcrypt unregistered [ 2418.178817] Key type ._llcrypt registered [ 2418.180574] Key type .llcrypt registered [ 2418.442717] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2418.485739] alg: No test for adler32 (adler32-zlib) [ 2419.446429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2419.474063] LNet: Added LNI 192.168.201.16@tcp [8/256/0/180] [ 2426.870527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.116@tcp [ 2427.076619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 2427.666497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.201.116@tcp [ 2427.877865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2427.922927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2427.974303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2428.022952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2428.060848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2428.102548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2428.148730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2428.186857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2428.213711] LNet: 121142:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2428.229454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2428.262202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2428.312265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2428.361938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2428.436926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2428.502772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2429.368843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2429.615640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2429.653338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2429.680526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2429.705123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2429.728087] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.116@tcp [ 2429.741568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2429.768085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2429.795748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2429.824427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2429.837834] LNet: 121260:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2429.852381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2429.896656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2429.944330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2429.985775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2430.017291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2430.062889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2430.653965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.201.116@tcp [ 2430.899329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2430.927867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2430.953242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2430.983180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2431.009888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2431.037601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2431.067809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2431.092715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2431.105557] LNet: 121377:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2431.121546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2431.147663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2431.177387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2431.203825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2431.231931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2431.256891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2431.840533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2432.050391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2432.075398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2432.115633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2432.153514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2432.179611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2432.204292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2432.233773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2432.257109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2432.276787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2432.310047] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2432.324339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2432.344837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2432.367593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2432.397331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2432.645086] LNet: 121540:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2432.649592] LNetError: 121540:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2432.660863] LNet: Removed LNI 192.168.201.16@tcp [ 2432.933725] Key type .llcrypt unregistered [ 2432.935617] Key type ._llcrypt unregistered [ 2449.816262] Key type ._llcrypt registered [ 2449.818297] Key type .llcrypt registered [ 2451.192565] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 21:05:32 (1778288732) [ 2451.532832] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2451.559519] alg: No test for adler32 (adler32-zlib) [ 2452.537913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2452.885575] LNet: 122428:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2453.448377] Key type .llcrypt unregistered [ 2453.450322] Key type ._llcrypt unregistered [ 2459.210593] Key type ._llcrypt registered [ 2459.213405] Key type .llcrypt registered [ 2459.421769] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2459.442339] alg: No test for adler32 (adler32-zlib) [ 2460.394384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2460.429350] LNet: Added LNI 192.168.201.16@tcp [8/256/0/180] [ 2467.739426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.116@tcp [ 2475.495784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2475.557808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2475.580960] LNet: Added LNI 192.168.201.16@tcp1 [8/256/0/180] [ 2476.403610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 2476.834562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.201.116@tcp [ 2477.024877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2477.056277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2477.089640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2477.115786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2477.138913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2477.167740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2477.198865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2477.237116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2477.254979] LNet: 123080:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2477.274050] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2477.303322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2477.326638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2477.352379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2477.375663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2477.402978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2477.800919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2477.857763] LNet: 2 peer NIs in recovery (showing 2): 192.168.201.116@tcp, 192.168.201.116@tcp1 [ 2477.990669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2478.021552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2478.055477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2478.081588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2478.104533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2478.131338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2478.188899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2478.233903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2478.254871] LNet: 123197:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2478.267624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2478.285786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2478.301472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2478.321255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2478.344937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2478.382201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2478.915986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.201.116@tcp [ 2479.107538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2479.128248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2479.158542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2479.181154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2479.200483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2479.220471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2479.267458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2479.307307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2479.322852] LNet: 123314:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2479.343953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2479.373531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2479.402857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2479.458058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2479.513592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2479.547903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2480.134404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2480.345464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2480.364422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2480.404749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2480.437661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2480.478992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2480.514104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2480.532485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2480.581150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2480.610576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2480.641668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2480.670444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2480.697752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2480.748604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2480.788213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2484.992300] LNet: 123512:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2484.995934] LNetError: 123512:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2485.002878] LNet: Removed LNI 192.168.201.16@tcp [ 2485.014740] LNet: Removed LNI 192.168.201.16@tcp1 [ 2485.392846] Key type .llcrypt unregistered [ 2485.396586] Key type ._llcrypt unregistered [ 2500.710750] Key type ._llcrypt registered [ 2500.712202] Key type .llcrypt registered [ 2501.950161] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 21:06:23 (1778288783) [ 2508.471535] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 21:06:29 (1778288789) [ 2515.103317] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 21:06:36 (1778288796) [ 2521.209474] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 21:06:42 (1778288802) [ 2527.155887] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 21:06:48 (1778288808) [ 2532.753067] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 21:06:54 (1778288814) [ 2539.163598] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 21:07:00 (1778288820) [ 2545.691269] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 21:07:07 (1778288827) [ 2552.849701] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 21:07:13 (1778288833) [ 2558.965671] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 21:07:20 (1778288840) [ 2565.021815] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 21:07:26 (1778288846) [ 2571.621494] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 21:07:32 (1778288852) [ 2577.392146] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 21:07:39 (1778288859) [ 2583.282631] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 21:07:44 (1778288864) [ 2588.759417] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 21:07:50 (1778288870) [ 2594.622291] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 21:07:56 (1778288876) [ 2599.734687] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 21:08:01 (1778288881) [ 2604.880163] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 21:08:06 (1778288886) [ 2605.233962] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2605.247570] alg: No test for adler32 (adler32-zlib) [ 2606.130357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2606.230674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2606.261052] LNet: Added LNI 192.168.201.16@tcp [8/256/0/180] [ 2607.296388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2607.314469] LNet: 134236:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2607.322712] LNetError: 134236:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2607.331299] LNet: Removed LNI 192.168.201.16@tcp [ 2607.347408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2607.402201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2607.426777] LNet: Added LNI 192.168.201.16@tcp [8/256/0/180] [ 2612.515605] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 21:08:14 (1778288894) [ 2618.049524] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 21:08:19 (1778288899) [ 2623.630690] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 21:08:25 (1778288905) [ 2628.436583] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 21:08:30 (1778288910) [ 2633.953940] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 21:08:35 (1778288915) [ 2639.201046] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 21:08:40 (1778288920) [ 2645.084508] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 21:08:46 (1778288926) [ 2651.702731] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 21:08:53 (1778288933) [ 2657.509345] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 21:08:58 (1778288938) [ 2664.033042] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 21:09:05 (1778288945) [ 2669.073454] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 21:09:10 (1778288950) [ 2674.339098] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 21:09:16 (1778288956) [ 2679.686972] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 2410 sec ========== 21:09:21 (1778288961) [ 2679.950537] LNet: 141697:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2679.956187] LNetError: 141697:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2679.971075] LNet: Removed LNI 192.168.201.16@tcp [ 2680.548209] Key type .llcrypt unregistered [ 2680.550921] Key type ._llcrypt unregistered [ 2690.069715] Key type ._llcrypt registered [ 2690.076830] Key type .llcrypt registered [ 2690.582179] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2690.598517] alg: No test for adler32 (adler32-zlib) [ 2691.687843] Lustre: Lustre: Build Version: 2.17.52_6_gcfb9284 [ 2691.898659] LNet: Added LNI 192.168.201.16@tcp [8/256/0/180] [ 2693.551171] Key type lgssc registered [ 2694.396644] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2738.527425] Lustre: Mounted lustre-client [ 2742.771507] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2753.664871] LustreError: 144147:0:(lov_obd.c:786:lov_cleanup()) lustre-clilov-ffff9775d85b3800: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 2753.706616] Lustre: Unmounted lustre-client [ 2772.497424] Key type lgssc unregistered [ 2772.717942] LNet: 144627:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2772.725704] LNetError: 144627:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2772.749557] LNet: Removed LNI 192.168.201.16@tcp [ 2773.161156] Key type .llcrypt unregistered [ 2773.163087] Key type ._llcrypt unregistered