[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 443817614 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.003091] x2apic enabled [ 0.004008] Switched APIC routing to physical x2apic. [ 0.005017] kvm-guest: setup PV IPIs [ 0.008000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008022] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009012] pid_max: default: 32768 minimum: 301 [ 0.010135] LSM: Security Framework initializing [ 0.011056] Yama: becoming mindful. [ 0.012039] SELinux: Initializing. [ 0.014070] *** VALIDATE selinux *** [ 0.022459] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027464] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.029079] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030105] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031100] *** VALIDATE tmpfs *** [ 0.032441] *** VALIDATE proc *** [ 0.033222] *** VALIDATE cgroup *** [ 0.034009] *** VALIDATE cgroup2 *** [ 0.035027] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.037095] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.038008] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.039028] Spectre V2 : User space: Vulnerable [ 0.040012] Speculative Store Bypass: Vulnerable [ 0.042920] debug: unmapping init [mem 0xffffffff99a59000-0xffffffff99a60fff] [ 0.045149] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.046744] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.047026] ... version: 2 [ 0.048000] ... bit width: 48 [ 0.048893] ... generic registers: 4 [ 0.049009] ... value mask: 0000ffffffffffff [ 0.050012] ... max period: 00007fffffffffff [ 0.051009] ... fixed-purpose events: 3 [ 0.051826] ... event mask: 000000070000000f [ 0.052327] rcu: Hierarchical SRCU implementation. [ 0.054519] smp: Bringing up secondary CPUs ... [ 0.055601] x86: Booting SMP configuration: [ 0.056020] .... node #0, CPUs: #1 #2 #3 [ 0.059082] smp: Brought up 1 node, 4 CPUs [ 0.060818] smpboot: Max logical packages: 1 [ 0.061013] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.135410] node 0 deferred pages initialised in 72ms [ 0.138470] devtmpfs: initialized [ 0.139155] x86/mm: Memory block size: 128MB [ 0.141848] gcov: version magic: 0x41383552 [ 0.143148] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.144085] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.145303] pinctrl core: initialized pinctrl subsystem [ 0.146170] [ 0.146447] ************************************************************* [ 0.148010] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.149008] ** ** [ 0.151010] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.152008] ** ** [ 0.153009] ** This means that this kernel is built to expose internal ** [ 0.155009] ** IOMMU data structures, which may compromise security on ** [ 0.156008] ** your system. ** [ 0.157010] ** ** [ 0.159009] ** If you see this message and you are not debugging the ** [ 0.160008] ** kernel, report this immediately to your vendor! ** [ 0.161009] ** ** [ 0.162010] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.164009] ************************************************************* [ 0.165616] NET: Registered protocol family 16 [ 0.167393] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.168050] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.169065] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.172012] cpuidle: using governor menu [ 0.173751] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.175386] PCI: Using configuration type 1 for base access [ 0.176130] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.184143] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.186020] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.189104] cryptd: max_cpu_qlen set to 1000 [ 0.192218] ACPI: Added _OSI(Module Device) [ 0.193013] ACPI: Added _OSI(Processor Device) [ 0.193921] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.195011] ACPI: Added _OSI(Processor Aggregator Device) [ 0.200270] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.205614] ACPI: Interpreter enabled [ 0.206060] ACPI: PM: (supports S0 S3 S4 S5) [ 0.206954] ACPI: Using IOAPIC for interrupt routing [ 0.208084] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.210362] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.220830] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.222037] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.224015] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.226072] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.230327] acpiphp: Slot [2] registered [ 0.231099] acpiphp: Slot [5] registered [ 0.231972] acpiphp: Slot [6] registered [ 0.233072] acpiphp: Slot [3] registered [ 0.233822] acpiphp: Slot [4] registered [ 0.234068] acpiphp: Slot [7] registered [ 0.234925] acpiphp: Slot [8] registered [ 0.236065] acpiphp: Slot [9] registered [ 0.236895] acpiphp: Slot [10] registered [ 0.238069] acpiphp: Slot [11] registered [ 0.239038] acpiphp: Slot [12] registered [ 0.239903] acpiphp: Slot [13] registered [ 0.241067] acpiphp: Slot [14] registered [ 0.241887] acpiphp: Slot [15] registered [ 0.242066] acpiphp: Slot [16] registered [ 0.242903] acpiphp: Slot [17] registered [ 0.244062] acpiphp: Slot [18] registered [ 0.244828] acpiphp: Slot [19] registered [ 0.245073] acpiphp: Slot [20] registered [ 0.246075] acpiphp: Slot [21] registered [ 0.246910] acpiphp: Slot [22] registered [ 0.248101] acpiphp: Slot [23] registered [ 0.249076] acpiphp: Slot [24] registered [ 0.249967] acpiphp: Slot [25] registered [ 0.251088] acpiphp: Slot [26] registered [ 0.251952] acpiphp: Slot [27] registered [ 0.253078] acpiphp: Slot [28] registered [ 0.254075] acpiphp: Slot [29] registered [ 0.255077] acpiphp: Slot [30] registered [ 0.256145] acpiphp: Slot [31] registered [ 0.257078] PCI host bridge to bus 0000:00 [ 0.259017] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.260015] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.261017] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.263029] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.264017] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.266019] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.267154] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.268913] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.271000] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.276469] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.281052] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.284019] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.285014] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.287019] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.289535] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.291776] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.294038] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.296818] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.300746] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.309013] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.313015] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.318068] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.323016] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.327014] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.338015] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.346771] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.351015] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.355013] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.365017] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.374578] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.377398] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.379315] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.381232] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.383212] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.389076] iommu: Default domain type: Passthrough [ 0.391199] SCSI subsystem initialized [ 0.392064] ACPI: bus type USB registered [ 0.392856] usbcore: registered new interface driver usbfs [ 0.394065] usbcore: registered new interface driver hub [ 0.395055] usbcore: registered new device driver usb [ 0.396103] pps_core: LinuxPPS API ver. 1 registered [ 0.397008] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.399057] PTP clock support registered [ 0.401061] EDAC MC: Ver: 3.0.0 [ 0.403128] PCI: Using ACPI for IRQ routing [ 0.404709] NetLabel: Initializing [ 0.405010] NetLabel: domain hash size = 128 [ 0.405925] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.407088] NetLabel: unlabeled traffic allowed by default [ 0.409084] vgaarb: loaded [ 0.411256] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.413010] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.418733] clocksource: Switched to clocksource kvm-clock [ 0.525281] VFS: Disk quotas dquot_6.6.0 [ 0.526611] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.528645] *** VALIDATE ramfs *** [ 0.529431] *** VALIDATE hugetlbfs *** [ 0.530462] pnp: PnP ACPI init [ 0.532386] pnp: PnP ACPI: found 6 devices [ 0.572271] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.574970] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.576916] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.578547] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.580048] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.581616] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.583369] NET: Registered protocol family 2 [ 0.585192] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.589367] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.591765] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.595556] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.597938] TCP: Hash tables configured (established 65536 bind 65536) [ 0.599951] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.602429] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.605139] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.608086] NET: Registered protocol family 1 [ 0.610347] RPC: Registered named UNIX socket transport module. [ 0.612172] RPC: Registered udp transport module. [ 0.613396] RPC: Registered tcp transport module. [ 0.614779] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.616765] NET: Registered protocol family 44 [ 0.618188] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.619788] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.621049] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.622631] PCI: CLS 0 bytes, default 64 [ 0.624566] Unpacking initramfs... [ 1.993408] debug: unmapping init [mem 0xffff90e5bcc64000-0xffff90e5bffcffff] [ 1.996858] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 1.999048] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.001893] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.501861] Initialise system trusted keyrings [ 2.502975] Key type blacklist registered [ 2.504252] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.511231] zbud: loaded [ 2.513969] *** VALIDATE nfs *** [ 2.515387] *** VALIDATE nfs4 *** [ 2.516881] pstore: using deflate compression [ 2.520072] Platform Keyring initialized [ 2.627094] NET: Registered protocol family 38 [ 2.628567] Key type asymmetric registered [ 2.629911] Asymmetric key parser 'x509' registered [ 2.631932] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.634953] io scheduler mq-deadline registered [ 2.636422] io scheduler kyber registered [ 2.637960] io scheduler bfq registered [ 2.639682] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.642520] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.645138] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.647780] ACPI: Power Button [PWRF] [ 2.653335] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.660367] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.686138] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.714647] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.743606] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.747594] Non-volatile memory driver v1.3 [ 2.748717] Linux agpgart interface v0.103 [ 2.778413] virtio_blk virtio1: [vda] 146640 512-byte logical blocks (75.1 MB/71.6 MiB) [ 2.780718] vda: detected capacity change from 0 to 75079680 [ 2.801354] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 2.804470] vdb: detected capacity change from 0 to 1073741824 [ 2.813303] libphy: Fixed MDIO Bus: probed [ 2.831988] usbcore: registered new interface driver usbserial_generic [ 2.834110] usbserial: USB Serial support registered for generic [ 2.835617] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 2.838293] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 2.839335] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 2.840772] mousedev: PS/2 mouse device common for all mice [ 2.842620] rtc_cmos 00:05: RTC can wake from S4 [ 2.844796] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 2.846390] rtc_cmos 00:05: registered as rtc0 [ 2.851865] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 2.853735] intel_pstate: CPU model not supported [ 2.856131] hid: raw HID events driver (C) Jiri Kosina [ 2.856767] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 2.857674] usbcore: registered new interface driver usbhid [ 2.861456] usbhid: USB HID core driver [ 2.862510] drop_monitor: Initializing network drop monitor service [ 2.864079] Initializing XFRM netlink socket [ 2.864322] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 2.865348] NET: Registered protocol family 10 [ 2.868927] Segment Routing with IPv6 [ 2.869752] NET: Registered protocol family 17 [ 2.871201] mpls_gso: MPLS GSO support [ 2.877073] RAS: Correctable Errors collector initialized. [ 2.878227] AVX version of gcm_enc/dec engaged. [ 2.879112] AES CTR mode by8 optimization enabled [ 2.967491] sched_clock: Marking stable (2967463757, 0)->(3787726963, -820263206) [ 2.970943] registered taskstats version 1 [ 2.972225] Loading compiled-in X.509 certificates [ 2.973494] zswap: loaded using pool lzo/zbud [ 2.996179] Key type big_key registered [ 3.011098] Key type encrypted registered [ 3.012484] ima: No TPM chip found, activating TPM-bypass! [ 3.014232] ima: Allocated hash algorithm: sha1 [ 3.015419] ima: No architecture policies found [ 3.016574] evm: Initialising EVM extended attributes: [ 3.017806] evm: security.selinux [ 3.018688] evm: security.ima [ 3.019407] evm: security.capability [ 3.020302] evm: HMAC attrs: 0x1 [ 3.022225] rtc_cmos 00:05: setting system clock to 2026-08-31 14:55:40 UTC (1788188140) [ 3.027459] debug: unmapping init [mem 0xffffffff9aa03000-0xffffffff9abfffff] [ 3.029527] debug: unmapping init [mem 0xffffffff99782000-0xffffffff99a58fff] [ 3.038222] Write protecting the kernel read-only data: 28672k [ 3.041467] debug: unmapping init [mem 0xffffffff97e03000-0xffffffff97ffffff] [ 3.044022] debug: unmapping init [mem 0xffffffff98714000-0xffffffff987fffff] [ 3.072978] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.079398] systemd[1]: Detected virtualization kvm. [ 3.080695] systemd[1]: Detected architecture x86-64. [ 3.081878] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.104548] systemd[1]: No hostname configured. [ 3.106079] systemd[1]: Set hostname to . [ 3.107985] random: systemd: uninitialized urandom read (16 bytes read) [ 3.110099] systemd[1]: Initializing machine ID from random generator. [ 3.153801] random: ln: uninitialized urandom read (6 bytes read) [ 3.240858] random: systemd: uninitialized urandom read (16 bytes read) [ 3.242851] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 3.246754] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 3.254306] systemd[1]: Starting Setup Virtual Console... Starting Setup Virtual Console... [ OK ] Listening on Journal Socket (/dev/log). Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Control Socket. [ OK ] Reached target Initrd Root Device. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. Starting Journal Service... [ OK ] Reached target Slices. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Timers. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Reached target Paths. Starting Apply Kernel Variables... [ OK ] Started Memstrack Anylazing Service. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 3.809513] device-mapper: uevent: version 1.0.3 [ 3.811710] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 4.564582] virtio_net virtio0 ens2: renamed from eth0 [ 4.636861] scsi host0: ata_piix [ 4.646332] scsi host1: ata_piix [ 4.659769] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 4.662369] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 9.065711] dracut-initqueue[595]: RTNETLINK answers: File exists [ 9.525091] random: crng init done [ 9.526462] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 9.957960] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Root Device. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Swap. Stopping udev Kernel Device Manager... [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Slices. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.192075] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.502449] SELinux: Disabled at runtime. [ 11.565139] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.573890] systemd[1]: Detected virtualization kvm. [ 11.575931] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.185229] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.188960] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.193923] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.197426] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.199832] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.208199] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.229703] systemd[1]: Listening on Process Core Dump Socket. [ OK ] Listening on Process Core Dump Socket. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Created slice system-getty.slice. Activating swap /dev/disk/by-label/SWAP... [ OK ] Started Dispatch Password Requests to Console Directory Watch. Starting Create list of required st…ce nodes for the current kernel... Mounting Huge Pages File System... [ 12.298732] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Reached target rpc_pipefs.target. Mounting Kernel Debug File System... Mounting POSIX Message Queue File System... Starting Apply Kernel Variables... Starting Remount Root and Kernel File Systems... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. [ OK ] Reached target Paths. [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Apply Kernel Variables. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ 12.654197] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 12.931232] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 12.966191] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.039944] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.054992] EDAC sbridge: Ver: 1.1.2 [ 14.069330] Key type dns_resolver registered [ 14.362164] NFS: Registering the id_resolver key type [ 14.364218] Key type id_resolver registered [ 14.365707] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started D-Bus System Message Bus. Starting Login Service... [ OK ] Started irqbalance daemon. Starting Network Manager... Starting Restore /run/initramfs on shutdown... [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Authorization Manager. [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg128-client login: [ 68.443544] libcfs: loading out-of-tree module taints kernel. [ 68.521294] Key type ._llcrypt registered [ 68.522474] Key type .llcrypt registered [ 69.069898] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 69.079709] alg: No test for adler32 (adler32-zlib) [ 70.656795] Lustre: Lustre: Build Version: 2.17.57_85_gb71fc30 [ 71.894512] LNet: Added LNI 192.168.201.28@tcp [8/256/0/180] [ 73.807183] Key type lgssc registered [ 75.889132] Lustre: Echo OBD driver; http://www.lustre.org/ [ 274.728556] Lustre: Mounted lustre-client - version 2.17.57_85_gb71fc30 [ 279.958960] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 295.537442] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing check_logdir /tmp/testlogs/ [ 300.511928] Lustre: lustre-OST0000-osc-ffff90e6057a4800: disconnect after 23s idle [ 303.277672] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing yml_node [ 311.329726] Lustre: DEBUG MARKER: Client: 2.17.57.85 [ 315.127139] hrtimer: interrupt took 15141944 ns [ 315.155892] Lustre: DEBUG MARKER: MDS: 2.17.57.85 [ 319.005504] Lustre: DEBUG MARKER: OSS: 2.17.57.85 [ 321.765284] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Mon Aug 31 11:00:57 EDT 2026 [ 344.639687] Lustre: Unmounted lustre-client [ 400.601429] Key type lgssc unregistered [ 400.906637] LNet: 9243:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 400.915700] LNetError: 9243:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 400.935253] LNet: Removed LNI 192.168.201.28@tcp [ 401.572294] Key type .llcrypt unregistered [ 401.574241] Key type ._llcrypt unregistered [ 439.913395] Key type ._llcrypt registered [ 439.915341] Key type .llcrypt registered [ 440.082589] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 440.093843] alg: No test for adler32 (adler32-zlib) [ 441.127581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 441.203399] LNet: Added LNI 192.168.201.28@tcp [8/256/0/180] [ 441.240774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 446.910687] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 447.008683] LNet: 10165:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 447.014452] LNetError: 10165:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 447.032844] LNet: Removed LNI 192.168.201.28@tcp [ 448.261555] Key type .llcrypt unregistered [ 448.268044] Key type ._llcrypt unregistered [ 458.096038] Key type ._llcrypt registered [ 458.102053] Key type .llcrypt registered [ 459.514964] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 11:03:15 (1788188595) [ 460.170801] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 460.189977] alg: No test for adler32 (adler32-zlib) [ 461.109680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 461.258789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 461.282630] LNet: 10586:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 461.312472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 461.373148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 461.393522] LNet: 10593:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 470.050830] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 11:03:25 (1788188605) [ 470.244637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 470.293614] LNet: 11180:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 470.458145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 470.661195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 483.095235] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 11:03:37 (1788188617) [ 483.293753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 483.303798] LNet: 11775:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 483.317178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 483.354211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 493.160328] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 11:03:48 (1788188628) [ 493.565820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 493.605994] LNet: 12369:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 493.639870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 493.770111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 502.318616] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 11:03:57 (1788188637) [ 502.443613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 502.481570] LNet: 12955:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 502.504839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 502.640958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 503.280898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 511.419327] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 11:04:06 (1788188646) [ 511.781281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 511.836260] LNet: 13560:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 511.869154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 511.945907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 512.670901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 513.383051] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 520.711836] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 11:04:16 (1788188656) [ 520.779655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 520.841847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 520.958265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 529.008809] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 11:04:24 (1788188664) [ 529.243837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 529.296492] LNet: 14758:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 529.308896] LNet: 14758:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 529.337748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 529.406405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 529.444668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 529.797615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 529.826397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 530.491473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 530.534479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 531.224818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 531.292880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 531.926671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 532.007438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 532.636285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 532.670390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 533.167496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 533.226335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 533.732880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 533.767681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 534.422918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 534.490616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 534.836392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 534.858712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 535.289284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 535.372619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 536.009645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 536.067377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 536.576969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 536.617906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 546.596140] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 11:04:41 (1788188681) [ 546.793276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 546.925630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 547.212490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 547.327922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 556.708867] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 11:04:52 (1788188692) [ 557.111117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 557.139232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 557.223840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 557.275434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 565.931914] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 11:05:01 (1788188701) [ 566.091178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 566.116783] LNet: 16631:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 566.125438] LNet: 16631:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 566.166285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 566.237262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 566.266190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 574.290702] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 11:05:10 (1788188710) [ 574.446835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 574.503979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 574.617907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 574.652535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 582.104543] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 11:05:17 (1788188717) [ 582.530967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 582.602513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 582.758541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 582.823614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 590.169167] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 11:05:26 (1788188726) [ 590.692837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 590.767063] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 590.876864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 590.964796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 598.698654] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 11:05:34 (1788188734) [ 598.903752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 598.960707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 599.135484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 599.859888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 608.387472] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 11:05:44 (1788188744) [ 608.600492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 608.657149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 608.824209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 609.010541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 616.450823] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 11:05:52 (1788188752) [ 616.822350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 616.853372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 617.030942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 617.318891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 624.687428] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 11:06:00 (1788188760) [ 625.151021] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 625.231247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 625.763673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 626.092651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 632.790136] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 11:06:08 (1788188768) [ 633.082225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 633.118239] LNet: 21532:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 633.127655] LNet: 21532:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 7 previous similar messages [ 633.180200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 633.293092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 633.377505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 640.552561] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 11:06:16 (1788188776) [ 640.781948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 640.836221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 640.914713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 640.960802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 651.212668] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 11:06:26 (1788188786) [ 651.389882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 651.490568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 651.628992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 659.211073] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 11:06:35 (1788188795) [ 659.546412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 659.579993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 659.640200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 667.721618] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 11:06:43 (1788188803) [ 668.072866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 668.117836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 668.252350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 668.843426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 676.728672] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 11:06:52 (1788188812) [ 677.016518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 677.052670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 677.111043] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 677.138381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 683.955850] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 11:06:59 (1788188819) [ 684.342430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 684.399618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 684.507426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 684.557363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 691.381822] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 11:07:07 (1788188827) [ 691.679657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 691.744986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 691.823540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 691.879064] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 699.738900] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 11:07:15 (1788188835) [ 699.971337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 700.006507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 700.062239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 700.099620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 706.962491] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 11:07:23 (1788188843) [ 707.448647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 707.533704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 707.646233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 707.740520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 707.761409] LNetError: 26920:0:(peer.c:2362:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 707.810590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 707.869915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 714.579312] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 11:07:30 (1788188850) [ 714.877430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 714.933483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 715.009762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.BBCYQMuCUj [ 715.131981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.BBCYQMuCUj [ 715.189786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.BBCYQMuCUj [ 721.526785] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 11:07:37 (1788188857) [ 721.732919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 721.771263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 721.831373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 721.862578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 721.944731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 728.199658] Lustre: DEBUG MARKER: == sanity-lnet test 40: LNetAddPeer merges a pre-existing non-primary peer NID ========================================================== 11:07:44 (1788188864) [ 728.450425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 728.485141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 728.544302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 728.560390] LNet: Added LNI 192.168.201.28@tcp [8/256/0/180] [ 729.818687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.28@tcp [ 731.144562] Lustre: Lustre: Build Version: 2.17.57_85_gb71fc30 [ 733.047362] Key type lgssc registered [ 734.519648] Lustre: Echo OBD driver; http://www.lustre.org/ [ 738.429252] LNetError: 28706:0:(lib-move.c:2158:lnet_handle_find_routed_path()) peer 192.168.201.28@tcp99 has no available nets [ 741.380769] Key type lgssc unregistered [ 741.783666] LNetError: 29838:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 741.808069] LNet: Removed LNI 192.168.201.28@tcp [ 742.372084] Key type .llcrypt unregistered [ 742.374414] Key type ._llcrypt unregistered [ 750.021060] Key type ._llcrypt registered [ 750.022769] Key type .llcrypt registered [ 751.558267] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 11:08:07 (1788188887) [ 752.116633] Key type .llcrypt unregistered [ 752.120397] Key type ._llcrypt unregistered [ 760.702529] Key type ._llcrypt registered [ 760.708489] Key type .llcrypt registered [ 761.000870] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 761.039367] alg: No test for adler32 (adler32-zlib) [ 762.026302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 762.075980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 762.101508] LNet: 30757:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=16 less than recommended minimum 512 [ 762.110777] LNet: 30757:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 [ 762.116506] LNet: 30757:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 [ 762.129797] LNet: Message forwarding will be enabled in 2s [ 765.215225] LNet: Message forwarding enabled [ 765.699379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 765.730662] LNet: Message forwarding will stop in 2s to 6s [ 769.313647] LNet: Message forwarding disabled [ 770.405670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 770.427730] LNet: 30856:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=16 less than recommended minimum 512 [ 770.440658] LNet: 30856:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 [ 770.456897] LNet: 30856:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 [ 770.479906] LNet: Message forwarding will be enabled in 2s [ 770.696174] LNet: 30869:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 771.467653] Key type .llcrypt unregistered [ 771.473764] Key type ._llcrypt unregistered [ 785.238815] Key type ._llcrypt registered [ 785.241309] Key type .llcrypt registered [ 786.951777] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 11:08:42 (1788188922) [ 787.544882] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 787.563383] alg: No test for adler32 (adler32-zlib) [ 788.548427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 788.655955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 797.208695] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 11:08:52 (1788188932) [ 797.383051] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 797.434429] LNet: 32283:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 797.468079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 797.602681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 797.706630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 797.764106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 797.840338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 797.902404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 797.965876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 798.051807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 798.118505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 798.197526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 798.280431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 798.359324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 798.442422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 798.517862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 798.592583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 798.693157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 798.777266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 798.836858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 798.864692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 798.896739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 798.910681] LNetError: 32330:0:(router.c:625:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 807.190607] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 11:09:02 (1788188942) [ 807.350533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 807.426889] LNet: 32902:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 807.510055] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 807.717131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 814.309880] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 11:09:10 (1788188950) [ 816.702978] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg128-server [ 819.198267] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 11:09:14 (1788188954) [ 819.414994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 819.430291] LNet: 33858:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 819.451233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 819.503303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 819.522606] LNet: Added LNI 192.168.201.28@tcp [8/256/0/180] [ 820.376708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.201.[29-33]@tcp [ 820.399086] LNet: 33992:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 820.465500] LNetError: 33862:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.201.29@tcp is being used as a gateway but routing feature is not turned on [ 820.478345] LNetError: 33862:0:(lib-lnet.h:1280:lnet_set_route_aliveness()) route to tcp1 through 192.168.201.29@tcp has gone from up to down [ 828.034928] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 11:09:23 (1788188963) [ 830.071537] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg128-server [ 832.553616] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 11:09:28 (1788188968) [ 833.006128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 833.033504] LNet: 34943:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 833.039329] LNetError: 34943:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 833.065064] LNet: Removed LNI 192.168.201.28@tcp [ 833.135669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 833.208395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 833.250202] LNet: Added LNI 192.168.201.28@tcp [8/256/0/180] [ 834.039579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.201.[29-34/2]@tcp [ 834.074613] LNet: 35060:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 834.079932] LNet: 35060:0:(router.c:729:lnet_add_route()) Skipped 4 previous similar messages [ 834.120528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.201.[29-34/2]@tcp [ 834.142181] LNetError: 34947:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.201.29@tcp is being used as a gateway but routing feature is not turned on [ 834.165139] LNetError: 34947:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 834.171868] LNetError: 34947:0:(lib-lnet.h:1280:lnet_set_route_aliveness()) route to tcp103 through 192.168.201.29@tcp has gone from up to down [ 834.180030] LNetError: 34947:0:(lib-lnet.h:1280:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 844.213556] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 11:09:39 (1788188979) [ 844.806878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 844.861396] LNet: 35661:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 844.895347] LNetError: 35661:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 844.919481] LNet: Removed LNI 192.168.201.28@tcp [ 844.984220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 845.141761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 845.187548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 845.254966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 845.319315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 845.363945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 845.400534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 845.464482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 845.574309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 845.603736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 845.709588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 845.821925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 845.904331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 846.018578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 846.137437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 846.186595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 846.367176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 846.492554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 846.594776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 846.687815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 846.823641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 846.853878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 846.902446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 846.963560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 846.996254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 847.049259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 847.110685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 847.174802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 847.239478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 847.312330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 847.350448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 847.448885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 847.541811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 847.595630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 847.696608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 854.989737] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 11:09:51 (1788188991) [ 856.933720] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg128-server [ 858.797519] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 11:09:54 (1788188994) [ 861.149667] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg128-server [ 863.244034] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 11:09:59 (1788188999) [ 863.408420] LNet: 37081:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 863.412241] LNet: 37081:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 11 previous similar messages [ 863.809238] Key type .llcrypt unregistered [ 863.812348] Key type ._llcrypt unregistered [ 869.702815] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 869.930164] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 869.942324] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 869.954845] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 871.286837] Key type ._llcrypt registered [ 871.288885] Key type .llcrypt registered [ 871.444274] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 871.460211] alg: No test for adler32 (adler32-zlib) [ 872.367367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 872.402290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 872.421916] LNet: Added LNI 192.168.201.28@tcp [8/256/0/180] [ 873.436336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 873.459655] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 876.515619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 876.619195] LNet: Removed LNI 10.1.2.3@tcp [ 876.870813] LNet: 37794:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 876.881095] LNetError: 37794:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 876.913216] LNet: Removed LNI 192.168.201.28@tcp [ 877.969649] Key type .llcrypt unregistered [ 877.973707] Key type ._llcrypt unregistered [ 891.139726] Key type ._llcrypt registered [ 891.141923] Key type .llcrypt registered [ 892.952550] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 11:10:28 (1788189028) [ 893.837415] Key type .llcrypt unregistered [ 893.843953] Key type ._llcrypt unregistered [ 901.382610] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 901.742873] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 901.749931] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 901.814421] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 905.237268] Key type ._llcrypt registered [ 905.261854] Key type .llcrypt registered [ 905.667291] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 905.684257] alg: No test for adler32 (adler32-zlib) [ 906.903500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 906.996246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 907.045102] LNet: Added LNI 192.168.201.28@tcp [8/256/0/180] [ 908.242485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 908.272474] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 912.792359] LNet: 39299:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 912.810096] LNetError: 39299:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 912.823793] LNet: Removed LNI 10.1.2.3@tcp [ 912.837075] LNet: Removed LNI 192.168.201.28@tcp [ 914.083598] Key type .llcrypt unregistered [ 914.085276] Key type ._llcrypt unregistered [ 930.681439] Key type ._llcrypt registered [ 930.691540] Key type .llcrypt registered [ 933.145943] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 11:11:07 (1788189067) [ 933.990454] Key type .llcrypt unregistered [ 934.001419] Key type ._llcrypt unregistered [ 942.150426] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 942.409366] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 942.430870] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 942.435534] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 944.044559] Key type ._llcrypt registered [ 944.051758] Key type .llcrypt registered [ 944.242498] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 944.272328] alg: No test for adler32 (adler32-zlib) [ 945.203767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 945.257936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 945.281518] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 947.291048] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 947.332569] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 949.758749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 949.778911] LNet: Removed LNI 10.1.2.3@tcp [ 949.793485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 949.835872] LNet: Removed LNI 10.1.2.31@tcp [ 949.887227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 949.931287] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 950.998452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 951.030673] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 952.018896] LNet: 41218:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 952.024302] LNetError: 41218:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg:0 not found [ 952.029850] LNet: Removed LNI 10.1.2.31@tcp [ 952.045902] LNet: Removed LNI 10.1.2.3@tcp [ 952.914908] Key type .llcrypt unregistered [ 952.916495] Key type ._llcrypt unregistered [ 967.467955] Key type ._llcrypt registered [ 967.469933] Key type .llcrypt registered [ 968.714849] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 11:11:44 (1788189104) [ 969.029321] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 969.041454] alg: No test for adler32 (adler32-zlib) [ 969.990718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 970.032849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 970.052353] LNet: Added LNI 192.168.201.28@tcp [8/256/0/180] [ 970.711377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 970.738926] LNet: 42178:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 970.744916] LNetError: 42178:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 970.761807] LNet: Removed LNI 192.168.201.28@tcp [ 970.775299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 970.832521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.201.28@tcp [ 970.855898] LNet: Added LNI 192.168.201.28@tcp [8/256/0/180] [ 971.640261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.201.28@tcp [ 971.671801] LNet: Removed LNI 192.168.201.28@tcp [ 971.764310] LNet: 42313:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 972.489630] Key type .llcrypt unregistered [ 972.492294] Key type ._llcrypt unregistered [ 985.090139] Key type ._llcrypt registered [ 985.093372] Key type .llcrypt registered [ 986.511889] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 11:12:02 (1788189122) [ 988.927509] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg128-server [ 991.321227] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 11:12:07 (1788189127) [ 991.971460] Key type .llcrypt unregistered [ 991.974521] Key type ._llcrypt unregistered [ 999.738193] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1000.000509] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1000.002821] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1000.009169] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1002.693726] Key type ._llcrypt registered [ 1002.695533] Key type .llcrypt registered [ 1002.831829] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1002.844975] alg: No test for adler32 (adler32-zlib) [ 1003.856983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1003.941264] LNet: Added LNI 192.168.201.28@tcp1 [8/256/0/180] [ 1004.903393] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 1007.831451] LNet: 44156:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1007.843361] LNetError: 44156:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1007.853353] LNet: Removed LNI 192.168.201.28@tcp1 [ 1007.871287] LNet: Removed LNI 10.1.2.3@tcp2 [ 1008.787679] Key type .llcrypt unregistered [ 1008.794837] Key type ._llcrypt unregistered [ 1023.444820] Key type ._llcrypt registered [ 1023.446529] Key type .llcrypt registered [ 1024.827430] Lustre: DEBUG MARKER: == sanity-lnet test 113: Test large skb Netlink dump with many LNet nets (LU-20221) ========================================================== 11:12:40 (1788189160) [ 1025.455364] Key type .llcrypt unregistered [ 1025.462237] Key type ._llcrypt unregistered [ 1033.011615] Key type ._llcrypt registered [ 1033.014199] Key type .llcrypt registered [ 1033.196204] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1033.218936] alg: No test for adler32 (adler32-zlib) [ 1034.157853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1034.225660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1034.261375] LNet: Added LNI 192.168.201.28@tcp1 [8/256/0/180] [ 1034.903738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1034.929318] LNet: Added LNI 192.168.201.28@tcp2 [8/256/0/180] [ 1035.841438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp3 --if ens2 [ 1035.871025] LNet: Added LNI 192.168.201.28@tcp3 [8/256/0/180] [ 1036.843667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp4 --if ens2 [ 1036.869602] LNet: Added LNI 192.168.201.28@tcp4 [8/256/0/180] [ 1037.629983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp5 --if ens2 [ 1037.648977] LNet: Added LNI 192.168.201.28@tcp5 [8/256/0/180] [ 1038.649987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp6 --if ens2 [ 1038.676100] LNet: Added LNI 192.168.201.28@tcp6 [8/256/0/180] [ 1040.151755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp7 --if ens2 [ 1040.182769] LNet: Added LNI 192.168.201.28@tcp7 [8/256/0/180] [ 1041.399917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp8 --if ens2 [ 1041.416621] LNet: Added LNI 192.168.201.28@tcp8 [8/256/0/180] [ 1042.710577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp9 --if ens2 [ 1042.727438] LNet: Added LNI 192.168.201.28@tcp9 [8/256/0/180] [ 1044.363515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [ 1044.399950] LNet: Added LNI 192.168.201.28@tcp10 [8/256/0/180] [ 1045.734661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [ 1045.768983] LNet: Added LNI 192.168.201.28@tcp11 [8/256/0/180] [ 1047.864555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [ 1047.879512] LNet: Added LNI 192.168.201.28@tcp12 [8/256/0/180] [ 1049.494280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [ 1049.514572] LNet: Added LNI 192.168.201.28@tcp13 [8/256/0/180] [ 1051.273233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [ 1051.297547] LNet: Added LNI 192.168.201.28@tcp14 [8/256/0/180] [ 1053.724253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [ 1053.742630] LNet: Added LNI 192.168.201.28@tcp15 [8/256/0/180] [ 1056.682516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [ 1056.731419] LNet: Added LNI 192.168.201.28@tcp16 [8/256/0/180] [ 1059.304117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [ 1059.338358] LNet: Added LNI 192.168.201.28@tcp17 [8/256/0/180] [ 1061.994678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [ 1062.028974] LNet: Added LNI 192.168.201.28@tcp18 [8/256/0/180] [ 1064.614114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [ 1064.639206] LNet: Added LNI 192.168.201.28@tcp19 [8/256/0/180] [ 1067.918683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [ 1067.935950] LNet: Added LNI 192.168.201.28@tcp20 [8/256/0/180] [ 1070.496339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [ 1070.524372] LNet: Added LNI 192.168.201.28@tcp21 [8/256/0/180] [ 1073.111754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [ 1073.143596] LNet: Added LNI 192.168.201.28@tcp22 [8/256/0/180] [ 1076.089869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [ 1076.113209] LNet: Added LNI 192.168.201.28@tcp23 [8/256/0/180] [ 1079.170286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [ 1079.192032] LNet: Added LNI 192.168.201.28@tcp24 [8/256/0/180] [ 1082.446313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [ 1082.474107] LNet: Added LNI 192.168.201.28@tcp25 [8/256/0/180] [ 1086.667928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [ 1086.684666] LNet: Added LNI 192.168.201.28@tcp26 [8/256/0/180] [ 1090.873485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [ 1090.902204] LNet: Added LNI 192.168.201.28@tcp27 [8/256/0/180] [ 1095.176858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [ 1095.220838] LNet: Added LNI 192.168.201.28@tcp28 [8/256/0/180] [ 1101.296508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [ 1101.313946] LNet: Added LNI 192.168.201.28@tcp29 [8/256/0/180] [ 1105.602454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [ 1105.624894] LNet: Added LNI 192.168.201.28@tcp30 [8/256/0/180] [ 1111.133120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [ 1111.155903] LNet: Added LNI 192.168.201.28@tcp31 [8/256/0/180] [ 1117.747589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [ 1117.771620] LNet: Added LNI 192.168.201.28@tcp32 [8/256/0/180] [ 1122.748935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [ 1122.774929] LNet: Added LNI 192.168.201.28@tcp33 [8/256/0/180] [ 1126.806682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [ 1126.829520] LNet: Added LNI 192.168.201.28@tcp34 [8/256/0/180] [ 1130.690991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [ 1130.711869] LNet: Added LNI 192.168.201.28@tcp35 [8/256/0/180] [ 1134.324575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [ 1134.348303] LNet: Added LNI 192.168.201.28@tcp36 [8/256/0/180] [ 1138.475719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [ 1138.493478] LNet: Added LNI 192.168.201.28@tcp37 [8/256/0/180] [ 1143.080850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [ 1143.095051] LNet: Added LNI 192.168.201.28@tcp38 [8/256/0/180] [ 1147.527210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [ 1147.562888] LNet: Added LNI 192.168.201.28@tcp39 [8/256/0/180] [ 1153.371596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [ 1153.410570] LNet: Added LNI 192.168.201.28@tcp40 [8/256/0/180] [ 1159.644701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [ 1159.665948] LNet: Added LNI 192.168.201.28@tcp41 [8/256/0/180] [ 1165.846710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [ 1165.884730] LNet: Added LNI 192.168.201.28@tcp42 [8/256/0/180] [ 1172.213705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [ 1172.239331] LNet: Added LNI 192.168.201.28@tcp43 [8/256/0/180] [ 1179.114535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [ 1179.157717] LNet: Added LNI 192.168.201.28@tcp44 [8/256/0/180] [ 1185.743496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [ 1185.758398] LNet: Added LNI 192.168.201.28@tcp45 [8/256/0/180] [ 1191.348636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [ 1191.410857] LNet: Added LNI 192.168.201.28@tcp46 [8/256/0/180] [ 1196.860944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [ 1196.887360] LNet: Added LNI 192.168.201.28@tcp47 [8/256/0/180] [ 1202.315268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [ 1202.352128] LNet: Added LNI 192.168.201.28@tcp48 [8/256/0/180] [ 1208.365549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [ 1208.392713] LNet: Added LNI 192.168.201.28@tcp49 [8/256/0/180] [ 1213.673400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [ 1213.689762] LNet: Added LNI 192.168.201.28@tcp50 [8/256/0/180] [ 1218.984221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp51 --if ens2 [ 1219.002400] LNet: Added LNI 192.168.201.28@tcp51 [8/256/0/180] [ 1224.694352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp52 --if ens2 [ 1224.709925] LNet: Added LNI 192.168.201.28@tcp52 [8/256/0/180] [ 1232.548800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp53 --if ens2 [ 1232.574113] LNet: Added LNI 192.168.201.28@tcp53 [8/256/0/180] [ 1240.658958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp54 --if ens2 [ 1240.695245] LNet: Added LNI 192.168.201.28@tcp54 [8/256/0/180] [ 1247.881165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp55 --if ens2 [ 1247.907533] LNet: Added LNI 192.168.201.28@tcp55 [8/256/0/180] [ 1256.534994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp56 --if ens2 [ 1256.591979] LNet: Added LNI 192.168.201.28@tcp56 [8/256/0/180] [ 1266.083960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp57 --if ens2 [ 1266.116667] LNet: Added LNI 192.168.201.28@tcp57 [8/256/0/180] [ 1274.820125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp58 --if ens2 [ 1274.843777] LNet: Added LNI 192.168.201.28@tcp58 [8/256/0/180] [ 1283.513809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp59 --if ens2 [ 1283.543851] LNet: Added LNI 192.168.201.28@tcp59 [8/256/0/180] [ 1292.908380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp60 --if ens2 [ 1292.945102] LNet: Added LNI 192.168.201.28@tcp60 [8/256/0/180] [ 1300.594964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp61 --if ens2 [ 1300.612361] LNet: Added LNI 192.168.201.28@tcp61 [8/256/0/180] [ 1307.165742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp62 --if ens2 [ 1307.190874] LNet: Added LNI 192.168.201.28@tcp62 [8/256/0/180] [ 1315.435224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp63 --if ens2 [ 1315.470391] LNet: Added LNI 192.168.201.28@tcp63 [8/256/0/180] [ 1326.750823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp64 --if ens2 [ 1326.771594] LNet: Added LNI 192.168.201.28@tcp64 [8/256/0/180] [ 1335.213822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp65 --if ens2 [ 1335.227117] LNet: Added LNI 192.168.201.28@tcp65 [8/256/0/180] [ 1342.580690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp66 --if ens2 [ 1342.597806] LNet: Added LNI 192.168.201.28@tcp66 [8/256/0/180] [ 1351.033697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp67 --if ens2 [ 1351.077613] LNet: Added LNI 192.168.201.28@tcp67 [8/256/0/180] [ 1360.594367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp68 --if ens2 [ 1360.656338] LNet: Added LNI 192.168.201.28@tcp68 [8/256/0/180] [ 1371.264565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp69 --if ens2 [ 1371.293099] LNet: Added LNI 192.168.201.28@tcp69 [8/256/0/180] [ 1380.429321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp70 --if ens2 [ 1380.478181] LNet: Added LNI 192.168.201.28@tcp70 [8/256/0/180] [ 1391.649687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp71 --if ens2 [ 1391.693226] LNet: Added LNI 192.168.201.28@tcp71 [8/256/0/180] [ 1406.693402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp72 --if ens2 [ 1406.711990] LNet: Added LNI 192.168.201.28@tcp72 [8/256/0/180] [ 1420.045945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp73 --if ens2 [ 1420.065818] LNet: Added LNI 192.168.201.28@tcp73 [8/256/0/180] [ 1428.942673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp74 --if ens2 [ 1428.963850] LNet: Added LNI 192.168.201.28@tcp74 [8/256/0/180] [ 1439.510761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp75 --if ens2 [ 1439.541853] LNet: Added LNI 192.168.201.28@tcp75 [8/256/0/180] [ 1448.832523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp76 --if ens2 [ 1448.860153] LNet: Added LNI 192.168.201.28@tcp76 [8/256/0/180] [ 1458.400861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp77 --if ens2 [ 1458.420444] LNet: Added LNI 192.168.201.28@tcp77 [8/256/0/180] [ 1467.930472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp78 --if ens2 [ 1467.941821] LNet: Added LNI 192.168.201.28@tcp78 [8/256/0/180] [ 1478.794968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp79 --if ens2 [ 1478.822763] LNet: Added LNI 192.168.201.28@tcp79 [8/256/0/180] [ 1490.435823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp80 --if ens2 [ 1490.461175] LNet: Added LNI 192.168.201.28@tcp80 [8/256/0/180] [ 1500.258077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp81 --if ens2 [ 1500.276854] LNet: Added LNI 192.168.201.28@tcp81 [8/256/0/180] [ 1510.228664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp82 --if ens2 [ 1510.252913] LNet: Added LNI 192.168.201.28@tcp82 [8/256/0/180] [ 1520.722174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp83 --if ens2 [ 1520.750146] LNet: Added LNI 192.168.201.28@tcp83 [8/256/0/180] [ 1530.892571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp84 --if ens2 [ 1530.928963] LNet: Added LNI 192.168.201.28@tcp84 [8/256/0/180] [ 1540.673872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp85 --if ens2 [ 1540.695093] LNet: Added LNI 192.168.201.28@tcp85 [8/256/0/180] [ 1550.478654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp86 --if ens2 [ 1550.503061] LNet: Added LNI 192.168.201.28@tcp86 [8/256/0/180] [ 1562.754383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp87 --if ens2 [ 1562.788939] LNet: Added LNI 192.168.201.28@tcp87 [8/256/0/180] [ 1577.504189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp88 --if ens2 [ 1577.525955] LNet: Added LNI 192.168.201.28@tcp88 [8/256/0/180] [ 1587.990825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp89 --if ens2 [ 1588.058992] LNet: Added LNI 192.168.201.28@tcp89 [8/256/0/180] [ 1602.899405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp90 --if ens2 [ 1602.931179] LNet: Added LNI 192.168.201.28@tcp90 [8/256/0/180] [ 1614.404402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp91 --if ens2 [ 1614.446691] LNet: Added LNI 192.168.201.28@tcp91 [8/256/0/180] [ 1625.746726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp92 --if ens2 [ 1625.783058] LNet: Added LNI 192.168.201.28@tcp92 [8/256/0/180] [ 1639.285966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp93 --if ens2 [ 1639.304273] LNet: Added LNI 192.168.201.28@tcp93 [8/256/0/180] [ 1654.879593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp94 --if ens2 [ 1654.901208] LNet: Added LNI 192.168.201.28@tcp94 [8/256/0/180] [ 1668.180698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp95 --if ens2 [ 1668.207436] LNet: Added LNI 192.168.201.28@tcp95 [8/256/0/180] [ 1686.130931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp96 --if ens2 [ 1686.176276] LNet: Added LNI 192.168.201.28@tcp96 [8/256/0/180] [ 1704.129915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp97 --if ens2 [ 1704.155480] LNet: Added LNI 192.168.201.28@tcp97 [8/256/0/180] [ 1717.476546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp98 --if ens2 [ 1717.498663] LNet: Added LNI 192.168.201.28@tcp98 [8/256/0/180] [ 1729.158845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if ens2 [ 1729.186426] LNet: Added LNI 192.168.201.28@tcp99 [8/256/0/180] [ 1739.172380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp100 --if ens2 [ 1739.190553] LNet: Added LNI 192.168.201.28@tcp100 [8/256/0/180] [ 1749.362893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp101 --if ens2 [ 1749.377912] LNet: Added LNI 192.168.201.28@tcp101 [8/256/0/180] [ 1760.141176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp102 --if ens2 [ 1760.158257] LNet: Added LNI 192.168.201.28@tcp102 [8/256/0/180] [ 1775.383100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp103 --if ens2 [ 1775.420726] LNet: Added LNI 192.168.201.28@tcp103 [8/256/0/180] [ 1789.417575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp104 --if ens2 [ 1789.440736] LNet: Added LNI 192.168.201.28@tcp104 [8/256/0/180] [ 1800.225163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp105 --if ens2 [ 1800.254435] LNet: Added LNI 192.168.201.28@tcp105 [8/256/0/180] [ 1812.834309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp106 --if ens2 [ 1812.846308] LNet: Added LNI 192.168.201.28@tcp106 [8/256/0/180] [ 1824.005692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp107 --if ens2 [ 1824.027144] LNet: Added LNI 192.168.201.28@tcp107 [8/256/0/180] [ 1837.862866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp108 --if ens2 [ 1837.901690] LNet: Added LNI 192.168.201.28@tcp108 [8/256/0/180] [ 1852.053552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp109 --if ens2 [ 1852.080530] LNet: Added LNI 192.168.201.28@tcp109 [8/256/0/180] [ 1863.122612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp110 --if ens2 [ 1863.138055] LNet: Added LNI 192.168.201.28@tcp110 [8/256/0/180] [ 1877.044591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp111 --if ens2 [ 1877.073946] LNet: Added LNI 192.168.201.28@tcp111 [8/256/0/180] [ 1890.616669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp112 --if ens2 [ 1890.638054] LNet: Added LNI 192.168.201.28@tcp112 [8/256/0/180] [ 1908.137157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp113 --if ens2 [ 1908.153866] LNet: Added LNI 192.168.201.28@tcp113 [8/256/0/180] [ 1921.871419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp114 --if ens2 [ 1921.916682] LNet: Added LNI 192.168.201.28@tcp114 [8/256/0/180] [ 1940.692226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp115 --if ens2 [ 1940.716418] LNet: Added LNI 192.168.201.28@tcp115 [8/256/0/180] [ 1955.273528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp116 --if ens2 [ 1955.312402] LNet: Added LNI 192.168.201.28@tcp116 [8/256/0/180] [ 1968.899390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp117 --if ens2 [ 1968.913359] LNet: Added LNI 192.168.201.28@tcp117 [8/256/0/180] [ 1980.434728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp118 --if ens2 [ 1980.476205] LNet: Added LNI 192.168.201.28@tcp118 [8/256/0/180] [ 1992.599826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp119 --if ens2 [ 1992.625459] LNet: Added LNI 192.168.201.28@tcp119 [8/256/0/180] [ 2004.748733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp120 --if ens2 [ 2004.766772] LNet: Added LNI 192.168.201.28@tcp120 [8/256/0/180] [ 2019.950254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp121 --if ens2 [ 2019.986429] LNet: Added LNI 192.168.201.28@tcp121 [8/256/0/180] [ 2034.388636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp122 --if ens2 [ 2034.425540] LNet: Added LNI 192.168.201.28@tcp122 [8/256/0/180] [ 2049.630203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp123 --if ens2 [ 2049.662264] LNet: Added LNI 192.168.201.28@tcp123 [8/256/0/180] [ 2065.911630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp124 --if ens2 [ 2065.934067] LNet: Added LNI 192.168.201.28@tcp124 [8/256/0/180] [ 2079.261301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp125 --if ens2 [ 2079.278935] LNet: Added LNI 192.168.201.28@tcp125 [8/256/0/180] [ 2093.090990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp126 --if ens2 [ 2093.105907] LNet: Added LNI 192.168.201.28@tcp126 [8/256/0/180] [ 2108.037183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp127 --if ens2 [ 2108.072171] LNet: Added LNI 192.168.201.28@tcp127 [8/256/0/180] [ 2133.901951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp128 --if ens2 [ 2133.917707] LNet: Added LNI 192.168.201.28@tcp128 [8/256/0/180] [ 2149.908521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp129 --if ens2 [ 2149.931814] LNet: Added LNI 192.168.201.28@tcp129 [8/256/0/180] [ 2164.363665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp130 --if ens2 [ 2164.383766] LNet: Added LNI 192.168.201.28@tcp130 [8/256/0/180] [ 2178.465095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp131 --if ens2 [ 2178.503499] LNet: Added LNI 192.168.201.28@tcp131 [8/256/0/180] [ 2193.240227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp132 --if ens2 [ 2193.256130] LNet: Added LNI 192.168.201.28@tcp132 [8/256/0/180] [ 2209.008353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp133 --if ens2 [ 2209.025082] LNet: Added LNI 192.168.201.28@tcp133 [8/256/0/180] [ 2229.448500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp134 --if ens2 [ 2229.472881] LNet: Added LNI 192.168.201.28@tcp134 [8/256/0/180] [ 2246.269707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp135 --if ens2 [ 2246.285063] LNet: Added LNI 192.168.201.28@tcp135 [8/256/0/180] [ 2268.641109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp136 --if ens2 [ 2268.649545] LNet: Added LNI 192.168.201.28@tcp136 [8/256/0/180] [ 2282.811410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp137 --if ens2 [ 2282.828808] LNet: Added LNI 192.168.201.28@tcp137 [8/256/0/180] [ 2298.582780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp138 --if ens2 [ 2298.600338] LNet: Added LNI 192.168.201.28@tcp138 [8/256/0/180] [ 2314.087882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp139 --if ens2 [ 2314.112420] LNet: Added LNI 192.168.201.28@tcp139 [8/256/0/180] [ 2330.050429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp140 --if ens2 [ 2330.097959] LNet: Added LNI 192.168.201.28@tcp140 [8/256/0/180] [ 2353.602582] LustreError: 168165:0:(udsp.c:674:lnet_udsp_apply_single_policy()) Bad action type. Expected 2 got 1 [ 2354.110810] LustreError: 168181:0:(udsp.c:674:lnet_udsp_apply_single_policy()) Bad action type. Expected 2 got 1 [ 2354.129457] LustreError: 168181:0:(udsp.c:674:lnet_udsp_apply_single_policy()) Skipped 50 previous similar messages [ 2355.090772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp1 --if ens2 [ 2355.123367] LNet: Removed LNI 192.168.201.28@tcp1 [ 2355.170874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp2 --if ens2 [ 2355.197831] LNet: Removed LNI 192.168.201.28@tcp2 [ 2355.253809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp3 --if ens2 [ 2355.304588] LNet: Removed LNI 192.168.201.28@tcp3 [ 2355.335655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp4 --if ens2 [ 2355.386509] LNet: Removed LNI 192.168.201.28@tcp4 [ 2355.430182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp5 --if ens2 [ 2355.456355] LNet: Removed LNI 192.168.201.28@tcp5 [ 2355.475554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp6 --if ens2 [ 2355.512412] LNet: Removed LNI 192.168.201.28@tcp6 [ 2355.533537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp7 --if ens2 [ 2355.585081] LNet: Removed LNI 192.168.201.28@tcp7 [ 2355.628654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp8 --if ens2 [ 2355.682247] LNet: Removed LNI 192.168.201.28@tcp8 [ 2355.723537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp9 --if ens2 [ 2355.770448] LNet: Removed LNI 192.168.201.28@tcp9 [ 2355.786563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp10 --if ens2 [ 2355.824680] LNet: Removed LNI 192.168.201.28@tcp10 [ 2355.873468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp11 --if ens2 [ 2355.906033] LNet: Removed LNI 192.168.201.28@tcp11 [ 2355.943965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp12 --if ens2 [ 2356.013584] LNet: Removed LNI 192.168.201.28@tcp12 [ 2356.048885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp13 --if ens2 [ 2356.087453] LNet: Removed LNI 192.168.201.28@tcp13 [ 2356.112549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp14 --if ens2 [ 2356.155810] LNet: Removed LNI 192.168.201.28@tcp14 [ 2356.214764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp15 --if ens2 [ 2356.252843] LNet: Removed LNI 192.168.201.28@tcp15 [ 2356.269656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp16 --if ens2 [ 2356.331425] LNet: Removed LNI 192.168.201.28@tcp16 [ 2356.352833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp17 --if ens2 [ 2356.382622] LNet: Removed LNI 192.168.201.28@tcp17 [ 2356.406730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp18 --if ens2 [ 2356.430403] LNet: Removed LNI 192.168.201.28@tcp18 [ 2356.466520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp19 --if ens2 [ 2356.512369] LNet: Removed LNI 192.168.201.28@tcp19 [ 2356.544736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp20 --if ens2 [ 2356.559410] LNet: Removed LNI 192.168.201.28@tcp20 [ 2356.610799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp21 --if ens2 [ 2356.645995] LNet: Removed LNI 192.168.201.28@tcp21 [ 2356.667845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp22 --if ens2 [ 2356.736790] LNet: Removed LNI 192.168.201.28@tcp22 [ 2356.775969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp23 --if ens2 [ 2356.838609] LNet: Removed LNI 192.168.201.28@tcp23 [ 2356.873450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp24 --if ens2 [ 2356.916559] LNet: Removed LNI 192.168.201.28@tcp24 [ 2356.944194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp25 --if ens2 [ 2356.996153] LNet: Removed LNI 192.168.201.28@tcp25 [ 2357.007908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp26 --if ens2 [ 2357.022895] LNet: Removed LNI 192.168.201.28@tcp26 [ 2357.040337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp27 --if ens2 [ 2357.070780] LNet: Removed LNI 192.168.201.28@tcp27 [ 2357.093594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp28 --if ens2 [ 2357.117424] LNet: Removed LNI 192.168.201.28@tcp28 [ 2357.136028] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp29 --if ens2 [ 2357.161967] LNet: Removed LNI 192.168.201.28@tcp29 [ 2357.194987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp30 --if ens2 [ 2357.233098] LNet: Removed LNI 192.168.201.28@tcp30 [ 2357.248994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp31 --if ens2 [ 2357.293552] LNet: Removed LNI 192.168.201.28@tcp31 [ 2357.314492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp32 --if ens2 [ 2357.343228] LNet: Removed LNI 192.168.201.28@tcp32 [ 2357.374562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp33 --if ens2 [ 2357.393992] LNet: Removed LNI 192.168.201.28@tcp33 [ 2357.411601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp34 --if ens2 [ 2357.440995] LNet: Removed LNI 192.168.201.28@tcp34 [ 2357.456237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp35 --if ens2 [ 2357.471941] LNet: Removed LNI 192.168.201.28@tcp35 [ 2357.487674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp36 --if ens2 [ 2357.509235] LNet: Removed LNI 192.168.201.28@tcp36 [ 2357.528749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp37 --if ens2 [ 2357.545355] LNet: Removed LNI 192.168.201.28@tcp37 [ 2357.564984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp38 --if ens2 [ 2357.589936] LNet: Removed LNI 192.168.201.28@tcp38 [ 2357.605698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp39 --if ens2 [ 2357.630779] LNet: Removed LNI 192.168.201.28@tcp39 [ 2357.644040] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp40 --if ens2 [ 2357.660198] LNet: Removed LNI 192.168.201.28@tcp40 [ 2357.678763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp41 --if ens2 [ 2357.696374] LNet: Removed LNI 192.168.201.28@tcp41 [ 2357.715858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp42 --if ens2 [ 2357.738521] LNet: Removed LNI 192.168.201.28@tcp42 [ 2357.756813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp43 --if ens2 [ 2357.770901] LNet: Removed LNI 192.168.201.28@tcp43 [ 2357.799822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp44 --if ens2 [ 2357.821896] LNet: Removed LNI 192.168.201.28@tcp44 [ 2357.841096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp45 --if ens2 [ 2357.877469] LNet: Removed LNI 192.168.201.28@tcp45 [ 2357.907131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp46 --if ens2 [ 2357.933814] LNet: Removed LNI 192.168.201.28@tcp46 [ 2357.946800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp47 --if ens2 [ 2357.981046] LNet: Removed LNI 192.168.201.28@tcp47 [ 2357.994963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp48 --if ens2 [ 2358.024645] LNet: Removed LNI 192.168.201.28@tcp48 [ 2358.045460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp49 --if ens2 [ 2358.070565] LNet: Removed LNI 192.168.201.28@tcp49 [ 2358.082474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp50 --if ens2 [ 2358.098538] LNet: Removed LNI 192.168.201.28@tcp50 [ 2358.113415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp51 --if ens2 [ 2358.123947] LNet: Removed LNI 192.168.201.28@tcp51 [ 2358.139996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp52 --if ens2 [ 2358.175331] LNet: Removed LNI 192.168.201.28@tcp52 [ 2358.202186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp53 --if ens2 [ 2358.261380] LNet: Removed LNI 192.168.201.28@tcp53 [ 2358.280892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp54 --if ens2 [ 2358.328786] LNet: Removed LNI 192.168.201.28@tcp54 [ 2358.370588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp55 --if ens2 [ 2358.421696] LNet: Removed LNI 192.168.201.28@tcp55 [ 2358.443710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp56 --if ens2 [ 2358.484663] LNet: Removed LNI 192.168.201.28@tcp56 [ 2358.526947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp57 --if ens2 [ 2358.583605] LNet: Removed LNI 192.168.201.28@tcp57 [ 2358.627557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp58 --if ens2 [ 2358.656460] LNet: Removed LNI 192.168.201.28@tcp58 [ 2358.688410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp59 --if ens2 [ 2358.728830] LNet: Removed LNI 192.168.201.28@tcp59 [ 2358.759517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp60 --if ens2 [ 2358.798872] LNet: Removed LNI 192.168.201.28@tcp60 [ 2358.830059] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp61 --if ens2 [ 2358.869646] LNet: Removed LNI 192.168.201.28@tcp61 [ 2358.903833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp62 --if ens2 [ 2358.925561] LNet: Removed LNI 192.168.201.28@tcp62 [ 2358.951826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp63 --if ens2 [ 2358.997939] LNet: Removed LNI 192.168.201.28@tcp63 [ 2359.036785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp64 --if ens2 [ 2359.083317] LNet: Removed LNI 192.168.201.28@tcp64 [ 2359.116845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp65 --if ens2 [ 2359.162694] LNet: Removed LNI 192.168.201.28@tcp65 [ 2359.197694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp66 --if ens2 [ 2359.233380] LNet: Removed LNI 192.168.201.28@tcp66 [ 2359.281824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp67 --if ens2 [ 2359.319828] LNet: Removed LNI 192.168.201.28@tcp67 [ 2359.366639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp68 --if ens2 [ 2359.412816] LNet: Removed LNI 192.168.201.28@tcp68 [ 2359.443850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp69 --if ens2 [ 2359.483806] LNet: Removed LNI 192.168.201.28@tcp69 [ 2359.515038] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp70 --if ens2 [ 2359.564866] LNet: Removed LNI 192.168.201.28@tcp70 [ 2359.603689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp71 --if ens2 [ 2359.633199] LNet: Removed LNI 192.168.201.28@tcp71 [ 2359.659691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp72 --if ens2 [ 2359.695091] LNet: Removed LNI 192.168.201.28@tcp72 [ 2359.727141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp73 --if ens2 [ 2359.763778] LNet: Removed LNI 192.168.201.28@tcp73 [ 2359.792489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp74 --if ens2 [ 2359.809610] LNet: Removed LNI 192.168.201.28@tcp74 [ 2359.844089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp75 --if ens2 [ 2359.876663] LNet: Removed LNI 192.168.201.28@tcp75 [ 2359.905840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp76 --if ens2 [ 2359.956854] LNet: Removed LNI 192.168.201.28@tcp76 [ 2359.983763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp77 --if ens2 [ 2360.019290] LNet: Removed LNI 192.168.201.28@tcp77 [ 2360.049426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp78 --if ens2 [ 2360.075402] LNet: Removed LNI 192.168.201.28@tcp78 [ 2360.088416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp79 --if ens2 [ 2360.109923] LNet: Removed LNI 192.168.201.28@tcp79 [ 2360.131568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp80 --if ens2 [ 2360.171378] LNet: Removed LNI 192.168.201.28@tcp80 [ 2360.205572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp81 --if ens2 [ 2360.253181] LNet: Removed LNI 192.168.201.28@tcp81 [ 2360.284958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp82 --if ens2 [ 2360.334662] LNet: Removed LNI 192.168.201.28@tcp82 [ 2360.358350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp83 --if ens2 [ 2360.400065] LNet: Removed LNI 192.168.201.28@tcp83 [ 2360.415426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp84 --if ens2 [ 2360.445809] LNet: Removed LNI 192.168.201.28@tcp84 [ 2360.506961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp85 --if ens2 [ 2360.555897] LNet: Removed LNI 192.168.201.28@tcp85 [ 2360.584445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp86 --if ens2 [ 2360.614895] LNet: Removed LNI 192.168.201.28@tcp86 [ 2360.636345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp87 --if ens2 [ 2360.657156] LNet: Removed LNI 192.168.201.28@tcp87 [ 2360.681587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp88 --if ens2 [ 2360.705149] LNet: Removed LNI 192.168.201.28@tcp88 [ 2360.731544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp89 --if ens2 [ 2360.762404] LNet: Removed LNI 192.168.201.28@tcp89 [ 2360.781405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp90 --if ens2 [ 2360.822943] LNet: Removed LNI 192.168.201.28@tcp90 [ 2360.847809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp91 --if ens2 [ 2360.875445] LNet: Removed LNI 192.168.201.28@tcp91 [ 2360.892616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp92 --if ens2 [ 2360.917386] LNet: Removed LNI 192.168.201.28@tcp92 [ 2360.933350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp93 --if ens2 [ 2360.954416] LNet: Removed LNI 192.168.201.28@tcp93 [ 2360.970909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp94 --if ens2 [ 2360.987922] LNet: Removed LNI 192.168.201.28@tcp94 [ 2361.009412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp95 --if ens2 [ 2361.031354] LNet: Removed LNI 192.168.201.28@tcp95 [ 2361.045856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp96 --if ens2 [ 2361.065527] LNet: Removed LNI 192.168.201.28@tcp96 [ 2361.085900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp97 --if ens2 [ 2361.116423] LNet: Removed LNI 192.168.201.28@tcp97 [ 2361.139352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp98 --if ens2 [ 2361.169693] LNet: Removed LNI 192.168.201.28@tcp98 [ 2361.190363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp99 --if ens2 [ 2361.218212] LNet: Removed LNI 192.168.201.28@tcp99 [ 2361.236491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp100 --if ens2 [ 2361.261848] LNet: Removed LNI 192.168.201.28@tcp100 [ 2361.282708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp101 --if ens2 [ 2361.327726] LNet: Removed LNI 192.168.201.28@tcp101 [ 2361.352447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp102 --if ens2 [ 2361.400996] LNet: Removed LNI 192.168.201.28@tcp102 [ 2361.457915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp103 --if ens2 [ 2361.493959] LNet: Removed LNI 192.168.201.28@tcp103 [ 2361.523139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp104 --if ens2 [ 2361.555754] LNet: Removed LNI 192.168.201.28@tcp104 [ 2361.593014] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp105 --if ens2 [ 2361.625786] LNet: Removed LNI 192.168.201.28@tcp105 [ 2361.649928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp106 --if ens2 [ 2361.668419] LNet: Removed LNI 192.168.201.28@tcp106 [ 2361.690947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp107 --if ens2 [ 2361.717370] LNet: Removed LNI 192.168.201.28@tcp107 [ 2361.743899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp108 --if ens2 [ 2361.779329] LNet: Removed LNI 192.168.201.28@tcp108 [ 2361.796901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp109 --if ens2 [ 2361.823790] LNet: Removed LNI 192.168.201.28@tcp109 [ 2361.868737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp110 --if ens2 [ 2361.898545] LNet: Removed LNI 192.168.201.28@tcp110 [ 2361.919765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp111 --if ens2 [ 2361.949541] LNet: Removed LNI 192.168.201.28@tcp111 [ 2361.998666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp112 --if ens2 [ 2362.025529] LNet: Removed LNI 192.168.201.28@tcp112 [ 2362.052923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp113 --if ens2 [ 2362.085037] LNet: Removed LNI 192.168.201.28@tcp113 [ 2362.109733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp114 --if ens2 [ 2362.153365] LNet: Removed LNI 192.168.201.28@tcp114 [ 2362.193833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp115 --if ens2 [ 2362.259898] LNet: Removed LNI 192.168.201.28@tcp115 [ 2362.298884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp116 --if ens2 [ 2362.338736] LNet: Removed LNI 192.168.201.28@tcp116 [ 2362.364882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp117 --if ens2 [ 2362.399386] LNet: Removed LNI 192.168.201.28@tcp117 [ 2362.423604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp118 --if ens2 [ 2362.457411] LNet: Removed LNI 192.168.201.28@tcp118 [ 2362.494391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp119 --if ens2 [ 2362.513406] LNet: Removed LNI 192.168.201.28@tcp119 [ 2362.534725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp120 --if ens2 [ 2362.586871] LNet: Removed LNI 192.168.201.28@tcp120 [ 2362.608302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp121 --if ens2 [ 2362.633648] LNet: Removed LNI 192.168.201.28@tcp121 [ 2362.662743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp122 --if ens2 [ 2362.701289] LNet: Removed LNI 192.168.201.28@tcp122 [ 2362.736930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp123 --if ens2 [ 2362.771114] LNet: Removed LNI 192.168.201.28@tcp123 [ 2362.812773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp124 --if ens2 [ 2362.842975] LNet: Removed LNI 192.168.201.28@tcp124 [ 2362.887620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp125 --if ens2 [ 2362.924378] LNet: Removed LNI 192.168.201.28@tcp125 [ 2362.972785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp126 --if ens2 [ 2363.022808] LNet: Removed LNI 192.168.201.28@tcp126 [ 2363.057791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp127 --if ens2 [ 2363.144740] LNet: Removed LNI 192.168.201.28@tcp127 [ 2363.194982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp128 --if ens2 [ 2363.266778] LNet: Removed LNI 192.168.201.28@tcp128 [ 2363.302323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp129 --if ens2 [ 2363.342864] LNet: Removed LNI 192.168.201.28@tcp129 [ 2363.396834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp130 --if ens2 [ 2363.474445] LNet: Removed LNI 192.168.201.28@tcp130 [ 2363.523344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp131 --if ens2 [ 2363.568353] LNet: Removed LNI 192.168.201.28@tcp131 [ 2363.589990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp132 --if ens2 [ 2363.623461] LNet: Removed LNI 192.168.201.28@tcp132 [ 2363.654949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp133 --if ens2 [ 2363.705918] LNet: Removed LNI 192.168.201.28@tcp133 [ 2363.745758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp134 --if ens2 [ 2363.788929] LNet: Removed LNI 192.168.201.28@tcp134 [ 2363.858799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp135 --if ens2 [ 2363.946900] LNet: Removed LNI 192.168.201.28@tcp135 [ 2364.045288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp136 --if ens2 [ 2364.113870] LNet: Removed LNI 192.168.201.28@tcp136 [ 2364.182638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp137 --if ens2 [ 2364.244134] LNet: Removed LNI 192.168.201.28@tcp137 [ 2364.274982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp138 --if ens2 [ 2364.308977] LNet: Removed LNI 192.168.201.28@tcp138 [ 2364.339215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp139 --if ens2 [ 2364.400353] LNet: Removed LNI 192.168.201.28@tcp139 [ 2364.433701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp140 --if ens2 [ 2364.531276] LNet: Removed LNI 192.168.201.28@tcp140 [ 2364.613742] LNet: 168498:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2365.458774] Key type .llcrypt unregistered [ 2365.465040] Key type ._llcrypt unregistered [ 2378.226381] Key type ._llcrypt registered [ 2378.234645] Key type .llcrypt registered [ 2380.020574] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 11:35:15 (1788190515) [ 2380.508501] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2380.519927] alg: No test for adler32 (adler32-zlib) [ 2381.414519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2381.452570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2381.472241] LNet: Added LNI 192.168.201.28@tcp [8/256/0/180] [ 2382.862374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2382.925164] LNet: 169465:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2382.934947] LNetError: 169465:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2382.950604] LNet: Removed LNI 192.168.201.28@tcp [ 2382.966666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2383.054568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 2383.079269] LNet: Added LNI 192.168.201.28@tcp [8/256/0/180] [ 2391.414303] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 11:35:27 (1788190527) [ 2391.937147] LNet: 170067:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2391.943397] LNetError: 170067:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2391.960596] LNet: Removed LNI 192.168.201.28@tcp [ 2393.002223] Key type .llcrypt unregistered [ 2393.005147] Key type ._llcrypt unregistered [ 2401.144338] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2401.299294] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2401.301402] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2401.316844] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2403.930430] Key type ._llcrypt registered [ 2403.932694] Key type .llcrypt registered [ 2404.198075] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2404.229183] alg: No test for adler32 (adler32-zlib) [ 2405.227615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2405.300480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2405.325287] LNet: Added LNI 192.168.201.28@tcp [8/256/0/180] [ 2406.089183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2406.113301] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2408.497859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2408.519695] LNet: 170808:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2408.528660] LNetError: 170808:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2408.535181] LNet: Removed LNI 10.1.2.3@tcp [ 2408.547420] LNet: Removed LNI 192.168.201.28@tcp [ 2408.565912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2408.620520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 2408.641585] LNet: Added LNI 192.168.201.28@tcp [8/256/0/180] [ 2408.644082] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2408.938639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2408.978400] LNet: Removed LNI 10.1.2.3@tcp [ 2408.986567] LNet: Removed LNI 192.168.201.28@tcp [ 2409.014211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2409.104777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 2409.144138] LNet: Added LNI 192.168.201.28@tcp [8/256/0/180] [ 2409.150528] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2409.760128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2409.788121] LNet: Added LNI 192.168.201.28@tcp2 [8/256/0/180] [ 2410.833593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2410.858844] LNet: 170997:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2410.867687] LNet: 170997:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 2410.883206] LNetError: 170997:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2410.894701] LNetError: 170997:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 2410.903968] LNet: Removed LNI 10.1.2.3@tcp [ 2410.906904] LNet: Removed LNI 192.168.201.28@tcp [ 2410.921811] LNet: Removed LNI 192.168.201.28@tcp2 [ 2410.939675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2411.011370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 2411.045159] LNet: Added LNI 192.168.201.28@tcp [8/256/0/180] [ 2411.050585] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2411.066362] LNet: Added LNI 192.168.201.28@tcp2 [8/256/0/180] [ 2419.111768] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 11:35:55 (1788190555) [ 2419.401547] LNet: 171604:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2419.419537] LNetError: 171604:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2419.428356] LNetError: 171604:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 2419.444184] LNet: Removed LNI 10.1.2.3@tcp [ 2419.450947] LNet: Removed LNI 192.168.201.28@tcp [ 2419.479026] LNet: Removed LNI 192.168.201.28@tcp2 [ 2420.164502] Key type .llcrypt unregistered [ 2420.167900] Key type ._llcrypt unregistered [ 2428.539270] Key type ._llcrypt registered [ 2428.541949] Key type .llcrypt registered [ 2428.724160] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2428.737547] alg: No test for adler32 (adler32-zlib) [ 2429.701978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2429.784480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2429.803973] LNet: Added LNI 192.168.201.28@tcp [8/256/0/180] [ 2430.788249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2430.811623] LNet: 171997:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2430.819810] LNetError: 171997:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2430.839612] LNet: Removed LNI 192.168.201.28@tcp [ 2430.907144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2431.027483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 2431.058351] LNet: Added LNI 192.168.201.28@tcp [8/256/0/180] [ 2431.805601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2431.844637] LNet: 172027:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2431.851473] LNetError: 172027:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2431.866341] LNet: Removed LNI 192.168.201.28@tcp [ 2431.887589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2431.934840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 2431.985597] LNet: Added LNI 192.168.201.28@tcp [8/256/0/180] [ 2432.687913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2432.720320] LNet: Removed LNI 192.168.201.28@tcp [ 2432.770579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2432.900988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 2432.936509] LNet: Added LNI 192.168.201.28@tcp [8/256/0/180] [ 2433.553721] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2433.723838] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2433.734730] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2433.746781] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2436.357536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2436.390707] LNet: 172136:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2436.398498] LNet: 172136:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 2436.407483] LNetError: 172136:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2436.419329] LNetError: 172136:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2436.439901] LNet: Removed LNI 192.168.201.28@tcp [ 2436.461717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2436.547297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 2436.591231] LNet: Added LNI 192.168.201.28@tcp [8/256/0/180] [ 2444.783898] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 11:36:20 (1788190580) [ 2444.876481] LNet: 172733:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2444.885856] LNetError: 172733:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2444.928225] LNet: Removed LNI 192.168.201.28@tcp [ 2445.596333] Key type .llcrypt unregistered [ 2445.597723] Key type ._llcrypt unregistered [ 2453.609664] Key type ._llcrypt registered [ 2453.617263] Key type .llcrypt registered [ 2453.912331] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2453.927950] alg: No test for adler32 (adler32-zlib) [ 2454.854354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2454.929090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2454.963982] LNet: Added LNI 192.168.201.28@tcp [8/256/0/180] [ 2455.920145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2455.944722] LNet: 173140:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2455.953650] LNetError: 173140:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2455.968291] LNet: Removed LNI 192.168.201.28@tcp [ 2455.984246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2456.039719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 2456.066231] LNet: Added LNI 192.168.201.28@tcp [8/256/0/180] [ 2463.425689] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 11:36:39 (1788190599) [ 2463.626593] LNet: 173740:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2463.634132] LNetError: 173740:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2463.644871] LNet: Removed LNI 192.168.201.28@tcp [ 2464.124296] Key type .llcrypt unregistered [ 2464.130856] Key type ._llcrypt unregistered [ 2471.092804] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2471.138732] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2471.146755] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2471.154742] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2473.711693] Key type ._llcrypt registered [ 2473.713910] Key type .llcrypt registered [ 2473.858707] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2473.871334] alg: No test for adler32 (adler32-zlib) [ 2474.746468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2474.786921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2474.812086] LNet: Added LNI 192.168.201.28@tcp [8/256/0/180] [ 2475.575873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2475.588314] LNet: 174190:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2475.592531] LNetError: 174190:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2475.602600] LNet: Removed LNI 192.168.201.28@tcp [ 2475.623297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2475.684394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 2475.700045] LNet: Added LNI 192.168.201.28@tcp [8/256/0/180] [ 2483.454534] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 11:36:59 (1788190619) [ 2483.615704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2483.674416] LNet: 174789:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2483.693133] LNetError: 174789:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2483.718976] LNet: Removed LNI 192.168.201.28@tcp [ 2483.754507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2483.976855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 2490.874514] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 11:37:07 (1788190627) [ 2491.123898] LNet: 175386:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2491.892972] Key type .llcrypt unregistered [ 2491.895235] Key type ._llcrypt unregistered [ 2499.433870] Key type ._llcrypt registered [ 2499.440160] Key type .llcrypt registered [ 2499.761085] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2499.805169] alg: No test for adler32 (adler32-zlib) [ 2500.793488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2500.868488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2500.894615] LNet: Added LNI 192.168.201.28@tcp [8/256/0/180] [ 2501.814933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2501.832786] LNet: 175782:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2501.841774] LNetError: 175782:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2501.857448] LNet: Removed LNI 192.168.201.28@tcp [ 2501.895849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2501.966593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 2502.017583] LNet: Added LNI 192.168.201.28@tcp [8/256/0/180] [ 2511.522584] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 11:37:27 (1788190647) [ 2511.878945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2511.896321] LNet: 176384:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2511.908418] LNetError: 176384:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2511.934514] LNet: Removed LNI 192.168.201.28@tcp [ 2511.969945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2512.112987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 2519.550847] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 11:37:35 (1788190655) [ 2521.419557] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 2523.574394] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 11:37:39 (1788190659) [ 2525.879629] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 2529.209450] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 11:37:44 (1788190664) [ 2531.454849] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 2534.017939] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 11:37:49 (1788190669) [ 2536.097863] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 2538.429710] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 11:37:54 (1788190674) [ 2540.547670] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 2542.663405] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing ========================================================== 11:37:58 (1788190678) [ 2543.189345] LNet: 178464:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2543.893897] Key type .llcrypt unregistered [ 2543.899745] Key type ._llcrypt unregistered [ 2550.328459] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2550.522364] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2550.558742] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2550.561350] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2553.355517] Key type ._llcrypt registered [ 2553.360451] Key type .llcrypt registered [ 2553.766344] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2553.800954] alg: No test for adler32 (adler32-zlib) [ 2554.999166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2555.166913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 2555.251803] LNet: Added LNI 192.168.201.28@tcp [8/512/0/180] [ 2555.325065] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 2555.336158] LNet: 178778:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 2555.465579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2555.476545] LNet: 178797:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2555.480987] LNetError: 178797:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2555.485465] LNet: Removed LNI 10.1.2.3@tcp [ 2555.487342] LNetError: 178797:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2555.510646] LNet: Removed LNI 192.168.201.28@tcp [ 2555.540225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2555.599267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 2555.615224] LNet: Added LNI 192.168.201.28@tcp [8/256/0/180] [ 2555.667860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2555.689458] LNet: 178821:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2555.694331] LNetError: 178821:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2555.709952] LNet: Removed LNI 192.168.201.28@tcp [ 2555.732372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2555.785517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 2555.803548] LNet: Added LNI 192.168.201.28@tcp [8/256/0/180] [ 2555.808067] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2555.810327] LNet: 178829:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 2555.951881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2555.975463] LNet: 178847:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2555.980648] LNetError: 178847:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2555.984919] LNet: Removed LNI 10.1.2.3@tcp [ 2555.987594] LNetError: 178847:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2556.000271] LNet: Removed LNI 192.168.201.28@tcp [ 2556.056891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2556.177893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 2556.239944] LNet: Added LNI 192.168.201.28@tcp [8/256/0/180] [ 2556.249175] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2556.251066] LNet: 178855:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 2556.509362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2556.534987] LNet: 178881:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2556.543919] LNetError: 178881:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2556.551705] LNet: Removed LNI 10.1.2.3@tcp [ 2556.553678] LNetError: 178881:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2556.577395] LNet: Removed LNI 192.168.201.28@tcp [ 2556.634163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2556.689209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 2556.715751] LNet: Added LNI 192.168.201.28@tcp [8/256/0/182] [ 2556.729780] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 2556.879696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2556.906126] LNet: 178909:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2556.913313] LNetError: 178909:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2556.918567] LNet: Removed LNI 10.1.2.3@tcp [ 2556.922243] LNetError: 178909:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2556.955496] LNet: Removed LNI 192.168.201.28@tcp [ 2556.990312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2557.101625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 2557.124155] LNet: Added LNI 192.168.201.28@tcp [8/256/0/182] [ 2557.129489] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 2557.133812] LNet: 178917:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 2557.191159] LNet: Removed LNI 10.1.2.3@tcp [ 2557.199241] LNet: Removed LNI 192.168.201.28@tcp [ 2558.133814] Key type .llcrypt unregistered [ 2558.141832] Key type ._llcrypt unregistered [ 2575.749861] Key type ._llcrypt registered [ 2575.752984] Key type .llcrypt registered [ 2577.049570] Lustre: DEBUG MARKER: == sanity-lnet test 180: FastReg map error should not leak descriptors ========================================================== 11:38:33 (1788190713) [ 2578.363499] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_180 Need o2ib NETTYPE [ 2580.516722] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 11:38:36 (1788190716) [ 2581.174933] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2581.190437] alg: No test for adler32 (adler32-zlib) [ 2582.247605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2582.352506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2582.406737] LNet: Added LNI 192.168.201.28@tcp [8/256/0/180] [ 2583.619741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.201.28 [ 2583.636246] LNet: 180194:0:(config.c:1738:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 2583.642486] LNet: Removed LNI 192.168.201.28@tcp [ 2591.118773] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 11:38:46 (1788190726) [ 2591.278392] LNet: 180766:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2591.294974] LNetError: 180766:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2591.315382] LNet: Removed LNI 192.168.201.28@tcp [ 2592.296556] Key type .llcrypt unregistered [ 2592.299153] Key type ._llcrypt unregistered [ 2599.928701] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2600.183800] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2602.855160] Key type ._llcrypt registered [ 2602.857357] Key type .llcrypt registered [ 2603.048089] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2603.072730] alg: No test for adler32 (adler32-zlib) [ 2604.297490] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2611.078585] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 11:39:07 (1788190747) [ 2611.532731] LNet: 181693:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2611.544320] LNetError: 181693:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2611.558407] LNet: Removed LNI 10.1.2.3@tcp [ 2612.185757] Key type .llcrypt unregistered [ 2612.187644] Key type ._llcrypt unregistered [ 2619.176404] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2619.527405] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2619.537875] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2619.553824] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2622.456788] Key type ._llcrypt registered [ 2622.459984] Key type .llcrypt registered [ 2622.658297] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2622.675894] alg: No test for adler32 (adler32-zlib) [ 2623.815139] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2632.370914] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 11:39:28 (1788190768) [ 2633.999584] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 2635.724430] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 11:39:31 (1788190771) [ 2636.009327] LNet: 182890:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2636.019577] LNetError: 182890:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2636.048471] LNet: Removed LNI 10.1.2.3@tcp [ 2636.567810] Key type .llcrypt unregistered [ 2636.569966] Key type ._llcrypt unregistered [ 2643.262543] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2643.516153] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2643.524339] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2643.531972] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2646.806476] Key type ._llcrypt registered [ 2646.808288] Key type .llcrypt registered [ 2647.029784] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2647.045631] alg: No test for adler32 (adler32-zlib) [ 2648.022793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2648.085096] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2657.302488] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 11:39:53 (1788190793) [ 2658.029033] LNet: 184129:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2658.038237] LNetError: 184129:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2658.056877] LNet: Removed LNI 10.1.2.3@tcp [ 2658.643475] Key type .llcrypt unregistered [ 2658.645268] Key type ._llcrypt unregistered [ 2665.424986] Key type ._llcrypt registered [ 2665.428694] Key type .llcrypt registered [ 2665.681499] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2665.712851] alg: No test for adler32 (adler32-zlib) [ 2666.700332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2666.731778] LNet: Added LNI 192.168.201.28@tcp [8/256/0/180] [ 2674.364633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.128@tcp [ 2675.155850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.28@tcp -d 192.168.201.128@tcp -m GET -e local_interrupt -r 1 [ 2675.229794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.28@tcp -d 192.168.201.28@tcp -m GET -e local_interrupt -r 1 [ 2675.307210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.128@tcp [ 2675.326720] LNet: There was an unexpected network error while writing to 192.168.201.128: rc = -22 [ 2675.630615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2675.672643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2675.726992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2675.775526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2675.819732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2675.871645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2675.908651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2675.935207] LNet: 1 local NIs in recovery (showing 1): 192.168.201.28@tcp [ 2675.952855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2675.972837] LNet: 184576:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2675.992314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2676.018681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2676.057929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2676.099816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2676.140997] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2676.194169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2676.245189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 2676.869111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.28@tcp -d 192.168.201.128@tcp -m GET -e local_dropped -r 1 [ 2676.901417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.28@tcp -d 192.168.201.28@tcp -m GET -e local_dropped -r 1 [ 2676.942412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.128@tcp [ 2676.964567] LNet: There was an unexpected network error while writing to 192.168.201.128: rc = -22 [ 2677.192507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2677.223372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2677.249867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2677.281232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2677.320847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2677.354571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2677.394302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2677.422687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2677.436887] LNet: 184685:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2677.452384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2677.501698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2677.561516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2677.597937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2677.642710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2677.684902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2677.731531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 2678.354049] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.28@tcp -d 192.168.201.128@tcp -m GET -e local_aborted -r 1 [ 2678.413904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.28@tcp -d 192.168.201.28@tcp -m GET -e local_aborted -r 1 [ 2678.446230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.128@tcp [ 2678.463887] LNet: There was an unexpected network error while writing to 192.168.201.128: rc = -22 [ 2678.687252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2678.723358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2678.764399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2678.796739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2678.831877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2678.889906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2678.948697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2678.986917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2679.006315] LNet: 184795:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2679.041517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2679.078762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2679.116661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2679.138440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2679.183308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2679.252340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2679.299645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 2679.828977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.28@tcp -d 192.168.201.128@tcp -m GET -e local_no_route -r 1 [ 2679.862258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.28@tcp -d 192.168.201.28@tcp -m GET -e local_no_route -r 1 [ 2679.888160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.128@tcp [ 2680.074903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2680.101904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2680.127835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2680.148387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2680.172842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2680.195200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2680.216659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2680.244887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2680.270333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2680.295850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2680.327437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2680.356567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2680.383074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2680.408450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2680.447270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 2680.903901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.28@tcp -d 192.168.201.128@tcp -m GET -e local_timeout -r 1 [ 2680.936651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.28@tcp -d 192.168.201.28@tcp -m GET -e local_timeout -r 1 [ 2680.966097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.128@tcp [ 2680.985510] LNet: There was an unexpected network error while writing to 192.168.201.128: rc = -22 [ 2680.990114] LNet: Skipped 2 previous similar messages [ 2681.193961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2681.217650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2681.238732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2681.264633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2681.292300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2681.318551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2681.341963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2681.362901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2681.381318] LNet: 185013:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2681.389680] LNet: 185013:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 2681.403419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2681.433901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2681.469817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2681.501553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2681.525983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2681.554711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2681.586675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 2682.049501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.28@tcp -d 192.168.201.128@tcp -m GET -e local_error -r 1 [ 2682.086764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.28@tcp -d 192.168.201.28@tcp -m GET -e local_error -r 1 [ 2682.121800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.128@tcp [ 2682.337670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2682.369400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2682.395866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2682.415167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2682.436790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2682.459300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2682.486696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2682.514883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2682.540808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2682.565752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2682.604182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2682.636700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2682.665858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2682.687392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2682.721740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 2682.978675] LNet: 185163:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2682.983321] LNetError: 185163:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2682.999449] LNet: Removed LNI 192.168.201.28@tcp [ 2683.332547] Key type .llcrypt unregistered [ 2683.334055] Key type ._llcrypt unregistered [ 2701.534253] Key type ._llcrypt registered [ 2701.536529] Key type .llcrypt registered [ 2702.700714] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 11:40:38 (1788190838) [ 2703.792178] Key type .llcrypt unregistered [ 2703.794601] Key type ._llcrypt unregistered [ 2709.128406] Key type ._llcrypt registered [ 2709.130344] Key type .llcrypt registered [ 2709.323299] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2709.370155] alg: No test for adler32 (adler32-zlib) [ 2710.291985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2710.314162] LNet: Added LNI 192.168.201.28@tcp [8/256/0/180] [ 2720.740351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.128@tcp [ 2732.710493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2732.805156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2732.862280] LNet: Added LNI 192.168.201.28@tcp1 [8/256/0/180] [ 2734.487843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.28@tcp -d 192.168.201.128@tcp -m GET -e local_interrupt -r 1 [ 2734.554246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.28@tcp -d 192.168.201.128@tcp1 -m GET -e local_interrupt -r 1 [ 2734.580479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.28@tcp -d 192.168.201.28@tcp -m GET -e local_interrupt -r 1 [ 2734.636681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.28@tcp -d 192.168.201.28@tcp1 -m GET -e local_interrupt -r 1 [ 2734.728424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.28@tcp1 -d 192.168.201.128@tcp -m GET -e local_interrupt -r 1 [ 2734.795375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.28@tcp1 -d 192.168.201.128@tcp1 -m GET -e local_interrupt -r 1 [ 2734.856817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.28@tcp1 -d 192.168.201.28@tcp -m GET -e local_interrupt -r 1 [ 2734.910125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.28@tcp1 -d 192.168.201.28@tcp1 -m GET -e local_interrupt -r 1 [ 2734.967133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.128@tcp [ 2734.982944] LNet: There was an unexpected network error while writing to 192.168.201.128: rc = -22 [ 2734.993814] LNet: 1 local NIs in recovery (showing 1): 192.168.201.28@tcp [ 2735.264860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2735.325729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2735.358873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2735.420143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2735.473118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2735.520423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2735.560148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2735.600599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2735.624397] LNet: 186642:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2735.645964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2735.699257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2735.751347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2735.842280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2735.918909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2736.013033] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2736.041381] LNet: There was an unexpected network error while writing to 192.168.201.28: rc = -22 [ 2736.058417] LNet: Skipped 3 previous similar messages [ 2736.086598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 2736.868970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.28@tcp -d 192.168.201.128@tcp -m GET -e local_dropped -r 1 [ 2736.909292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.28@tcp -d 192.168.201.128@tcp1 -m GET -e local_dropped -r 1 [ 2736.942319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.28@tcp -d 192.168.201.28@tcp -m GET -e local_dropped -r 1 [ 2736.997682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.28@tcp -d 192.168.201.28@tcp1 -m GET -e local_dropped -r 1 [ 2737.033344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.28@tcp1 -d 192.168.201.128@tcp -m GET -e local_dropped -r 1 [ 2737.075721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.28@tcp1 -d 192.168.201.128@tcp1 -m GET -e local_dropped -r 1 [ 2737.144290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.28@tcp1 -d 192.168.201.28@tcp -m GET -e local_dropped -r 1 [ 2737.205499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.28@tcp1 -d 192.168.201.28@tcp1 -m GET -e local_dropped -r 1 [ 2737.252414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.128@tcp [ 2737.281676] LNet: There was an unexpected network error while writing to 192.168.201.128: rc = -22 [ 2737.584964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2737.626576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2737.660353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2737.688976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2737.737964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2737.788057] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2737.845959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2737.897334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2737.928791] LNet: 186763:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2737.950770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2738.003321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2738.071379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2738.135445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2738.176379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2738.252951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2738.325391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 2739.304996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.28@tcp -d 192.168.201.128@tcp -m GET -e local_aborted -r 1 [ 2739.389697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.28@tcp -d 192.168.201.128@tcp1 -m GET -e local_aborted -r 1 [ 2739.428577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.28@tcp -d 192.168.201.28@tcp -m GET -e local_aborted -r 1 [ 2739.487453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.28@tcp -d 192.168.201.28@tcp1 -m GET -e local_aborted -r 1 [ 2739.571672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.28@tcp1 -d 192.168.201.128@tcp -m GET -e local_aborted -r 1 [ 2739.640736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.28@tcp1 -d 192.168.201.128@tcp1 -m GET -e local_aborted -r 1 [ 2739.705664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.28@tcp1 -d 192.168.201.28@tcp -m GET -e local_aborted -r 1 [ 2739.741344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.28@tcp1 -d 192.168.201.28@tcp1 -m GET -e local_aborted -r 1 [ 2739.782994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.128@tcp [ 2739.823303] LNet: There was an unexpected network error while writing to 192.168.201.128: rc = -22 [ 2739.830157] LNet: Skipped 2 previous similar messages [ 2740.140288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2740.179996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2740.217351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2740.289976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2740.369084] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2740.449602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2740.505122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2740.561855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2740.588659] LNet: 186885:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2740.628094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2740.685167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2740.748328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2740.788309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2740.838797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2740.909784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2740.960816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 2741.821210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.28@tcp -d 192.168.201.128@tcp -m GET -e local_no_route -r 1 [ 2741.898642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.28@tcp -d 192.168.201.128@tcp1 -m GET -e local_no_route -r 1 [ 2742.010763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.28@tcp -d 192.168.201.28@tcp -m GET -e local_no_route -r 1 [ 2742.110554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.28@tcp -d 192.168.201.28@tcp1 -m GET -e local_no_route -r 1 [ 2742.185774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.28@tcp1 -d 192.168.201.128@tcp -m GET -e local_no_route -r 1 [ 2742.247588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.28@tcp1 -d 192.168.201.128@tcp1 -m GET -e local_no_route -r 1 [ 2742.303696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.28@tcp1 -d 192.168.201.28@tcp -m GET -e local_no_route -r 1 [ 2742.402491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.28@tcp1 -d 192.168.201.28@tcp1 -m GET -e local_no_route -r 1 [ 2742.461697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.128@tcp [ 2742.875867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2742.937099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2742.995334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2743.039915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2743.085936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2743.165792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2743.225128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2743.304876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2743.338332] LNet: 187006:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2743.373328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2743.416359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2743.497830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2743.542564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2743.615895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2743.677650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2743.744284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 2744.660929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.28@tcp -d 192.168.201.128@tcp -m GET -e local_timeout -r 1 [ 2744.697492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.28@tcp -d 192.168.201.128@tcp1 -m GET -e local_timeout -r 1 [ 2744.760192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.28@tcp -d 192.168.201.28@tcp -m GET -e local_timeout -r 1 [ 2744.796750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.28@tcp -d 192.168.201.28@tcp1 -m GET -e local_timeout -r 1 [ 2744.848980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.28@tcp1 -d 192.168.201.128@tcp -m GET -e local_timeout -r 1 [ 2744.916058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.28@tcp1 -d 192.168.201.128@tcp1 -m GET -e local_timeout -r 1 [ 2744.996285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.28@tcp1 -d 192.168.201.28@tcp -m GET -e local_timeout -r 1 [ 2745.084902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.28@tcp1 -d 192.168.201.28@tcp1 -m GET -e local_timeout -r 1 [ 2745.125238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.128@tcp [ 2745.157247] LNet: There was an unexpected network error while writing to 192.168.201.128: rc = -22 [ 2745.169481] LNet: Skipped 5 previous similar messages [ 2745.586546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2745.655773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2745.713965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2745.757174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2745.812649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2745.899704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2745.966529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2746.032827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2746.079472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2746.143894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2746.226871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2746.301832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2746.349390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2746.408617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2746.445460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 2747.272428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.28@tcp -d 192.168.201.128@tcp -m GET -e local_error -r 1 [ 2747.324793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.28@tcp -d 192.168.201.128@tcp1 -m GET -e local_error -r 1 [ 2747.392084] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.28@tcp -d 192.168.201.28@tcp -m GET -e local_error -r 1 [ 2747.443778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.28@tcp -d 192.168.201.28@tcp1 -m GET -e local_error -r 1 [ 2747.501164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.28@tcp1 -d 192.168.201.128@tcp -m GET -e local_error -r 1 [ 2747.554821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.28@tcp1 -d 192.168.201.128@tcp1 -m GET -e local_error -r 1 [ 2747.659899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.28@tcp1 -d 192.168.201.28@tcp -m GET -e local_error -r 1 [ 2747.716760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.28@tcp1 -d 192.168.201.28@tcp1 -m GET -e local_error -r 1 [ 2747.825641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.128@tcp [ 2748.264724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2748.312922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2748.367867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2748.407255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2748.510934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2748.545846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2748.593803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2748.640652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2748.670619] LNet: 187249:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2748.679702] LNet: 187249:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 2748.719477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2748.744323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2748.774644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2748.849717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2748.906595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2748.961707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2749.013368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 2755.870480] LNet: 187326:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2755.879923] LNetError: 187326:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2755.887846] LNet: Removed LNI 192.168.201.28@tcp [ 2756.902615] LNet: Removed LNI 192.168.201.28@tcp1 [ 2757.400609] Key type .llcrypt unregistered [ 2757.402727] Key type ._llcrypt unregistered [ 2776.507927] Key type ._llcrypt registered [ 2776.510638] Key type .llcrypt registered [ 2777.785072] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 11:41:53 (1788190913) [ 2778.731595] Key type .llcrypt unregistered [ 2778.736741] Key type ._llcrypt unregistered [ 2785.499122] Key type ._llcrypt registered [ 2785.501500] Key type .llcrypt registered [ 2785.695872] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2785.725295] alg: No test for adler32 (adler32-zlib) [ 2786.692260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2786.750795] LNet: Added LNI 192.168.201.28@tcp [8/256/0/180] [ 2796.391340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.128@tcp [ 2797.640963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.28@tcp -d 192.168.201.128@tcp -m GET -e remote_dropped -r 1 [ 2797.751939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.28@tcp -d 192.168.201.28@tcp -m GET -e remote_dropped -r 1 [ 2797.831789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.128@tcp [ 2797.868354] LNet: There was an unexpected network error while writing to 192.168.201.128: rc = -22 [ 2798.296140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2798.363911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2798.453292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2798.549763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2798.605301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2798.690683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2798.748807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2798.817426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2798.856392] LNet: 188587:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2798.880470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2798.930945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2798.989139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2799.040730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2799.116086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2799.188861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2799.261306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 2800.229592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.28@tcp -d 192.168.201.128@tcp -m GET -e remote_error -r 1 [ 2800.306878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.28@tcp -d 192.168.201.28@tcp -m GET -e remote_error -r 1 [ 2800.384993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.128@tcp [ 2800.426170] LNet: There was an unexpected network error while writing to 192.168.201.128: rc = -22 [ 2800.710418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2800.754585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2800.805500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2800.859177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2800.905887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2800.953239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2800.991318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2801.025568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2801.048967] LNet: 188696:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2801.075127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2801.112262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2801.138916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2801.166519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2801.190821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2801.219068] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2801.245531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 2801.702422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.28@tcp -d 192.168.201.128@tcp -m GET -e remote_timeout -r 1 [ 2801.730298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.28@tcp -d 192.168.201.28@tcp -m GET -e remote_timeout -r 1 [ 2801.765767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.128@tcp [ 2801.783209] LNet: There was an unexpected network error while writing to 192.168.201.128: rc = -22 [ 2802.066965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2802.105602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2802.141433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2802.169522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2802.193987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2802.228849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2802.252748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2802.271497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2802.280285] LNet: 188806:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2802.290817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2802.321964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2802.352444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2802.388262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2802.417629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2802.443513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2802.472494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 2802.679857] LNet: 188847:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2802.687314] LNetError: 188847:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2802.705410] LNet: Removed LNI 192.168.201.28@tcp [ 2803.175555] Key type .llcrypt unregistered [ 2803.179121] Key type ._llcrypt unregistered [ 2824.319084] Key type ._llcrypt registered [ 2824.320699] Key type .llcrypt registered [ 2826.115746] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 11:42:41 (1788190961) [ 2827.119394] Key type .llcrypt unregistered [ 2827.123876] Key type ._llcrypt unregistered [ 2835.192283] Key type ._llcrypt registered [ 2835.195923] Key type .llcrypt registered [ 2835.564994] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2835.595759] alg: No test for adler32 (adler32-zlib) [ 2836.602981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2836.654859] LNet: Added LNI 192.168.201.28@tcp [8/256/0/180] [ 2845.802373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.128@tcp [ 2856.505242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2856.530698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2856.549824] LNet: Added LNI 192.168.201.28@tcp1 [8/256/0/180] [ 2858.241567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.28@tcp -d 192.168.201.128@tcp -m GET -e remote_dropped -r 1 [ 2858.268431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.28@tcp -d 192.168.201.128@tcp1 -m GET -e remote_dropped -r 1 [ 2858.312662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.28@tcp -d 192.168.201.28@tcp -m GET -e remote_dropped -r 1 [ 2858.358878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.28@tcp -d 192.168.201.28@tcp1 -m GET -e remote_dropped -r 1 [ 2858.424860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.28@tcp1 -d 192.168.201.128@tcp -m GET -e remote_dropped -r 1 [ 2858.506675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.28@tcp1 -d 192.168.201.128@tcp1 -m GET -e remote_dropped -r 1 [ 2858.579620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.28@tcp1 -d 192.168.201.28@tcp -m GET -e remote_dropped -r 1 [ 2858.659410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.28@tcp1 -d 192.168.201.28@tcp1 -m GET -e remote_dropped -r 1 [ 2858.714544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.128@tcp [ 2858.747109] LNet: There was an unexpected network error while writing to 192.168.201.128: rc = -22 [ 2858.763933] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.128@tcp [ 2859.179619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2859.235812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2859.283968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2859.360370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2859.488940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2859.578578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2859.661992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2859.752086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2859.777058] LNet: 190326:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2859.814717] LNet: There was an unexpected network error while writing to 192.168.201.128: rc = -22 [ 2859.829740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2859.830902] LNet: Skipped 3 previous similar messages [ 2859.903664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2859.948904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2860.008304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2860.063853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2860.140914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2860.232460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 2860.293583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2861.205615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.28@tcp -d 192.168.201.128@tcp -m GET -e remote_error -r 1 [ 2861.276913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.28@tcp -d 192.168.201.128@tcp1 -m GET -e remote_error -r 1 [ 2861.367668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.28@tcp -d 192.168.201.28@tcp -m GET -e remote_error -r 1 [ 2861.458530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.28@tcp -d 192.168.201.28@tcp1 -m GET -e remote_error -r 1 [ 2861.507535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.28@tcp1 -d 192.168.201.128@tcp -m GET -e remote_error -r 1 [ 2861.556211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.28@tcp1 -d 192.168.201.128@tcp1 -m GET -e remote_error -r 1 [ 2861.601795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.28@tcp1 -d 192.168.201.28@tcp -m GET -e remote_error -r 1 [ 2861.652319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.28@tcp1 -d 192.168.201.28@tcp1 -m GET -e remote_error -r 1 [ 2861.717761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.128@tcp [ 2861.748380] LNet: There was an unexpected network error while writing to 192.168.201.128: rc = -22 [ 2862.199069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2862.298715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2862.397607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2862.502699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2862.604645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2862.695896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2862.756956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2862.839804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2862.895076] LNet: 190450:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2862.942780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2863.007963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2863.120465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2863.167395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2863.244176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2863.323719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2863.465130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 2863.580735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2864.574325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.28@tcp -d 192.168.201.128@tcp -m GET -e remote_timeout -r 1 [ 2864.662713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.28@tcp -d 192.168.201.128@tcp1 -m GET -e remote_timeout -r 1 [ 2864.715524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.28@tcp -d 192.168.201.28@tcp -m GET -e remote_timeout -r 1 [ 2864.788420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.28@tcp -d 192.168.201.28@tcp1 -m GET -e remote_timeout -r 1 [ 2864.877508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.28@tcp1 -d 192.168.201.128@tcp -m GET -e remote_timeout -r 1 [ 2864.963964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.28@tcp1 -d 192.168.201.128@tcp1 -m GET -e remote_timeout -r 1 [ 2865.013584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.28@tcp1 -d 192.168.201.28@tcp -m GET -e remote_timeout -r 1 [ 2865.036186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.28@tcp1 -d 192.168.201.28@tcp1 -m GET -e remote_timeout -r 1 [ 2865.052825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.128@tcp [ 2865.064827] LNet: There was an unexpected network error while writing to 192.168.201.128: rc = -22 [ 2865.200842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2865.213955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2865.231619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2865.257338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2865.275867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2865.293858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2865.312344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2865.327179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2865.338882] LNet: 190573:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2865.351430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2865.374910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2865.393481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2865.421245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2865.434964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2865.448383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2865.470315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 2865.534841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2871.263091] LNet: 190651:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2871.272909] LNetError: 190651:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2871.300871] LNet: Removed LNI 192.168.201.28@tcp [ 2871.325282] LNet: Removed LNI 192.168.201.28@tcp1 [ 2872.023631] Key type .llcrypt unregistered [ 2872.025256] Key type ._llcrypt unregistered [ 2894.668364] Key type ._llcrypt registered [ 2894.669704] Key type .llcrypt registered [ 2896.520294] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 11:43:52 (1788191032) [ 2898.932356] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 2902.068197] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 11:43:57 (1788191037) [ 2903.374744] Key type .llcrypt unregistered [ 2903.379822] Key type ._llcrypt unregistered [ 2913.373735] Key type ._llcrypt registered [ 2913.375404] Key type .llcrypt registered [ 2913.496613] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2913.506544] alg: No test for adler32 (adler32-zlib) [ 2914.422854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2914.447757] LNet: Added LNI 192.168.201.28@tcp [8/256/0/180] [ 2923.559079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.128@tcp [ 2924.319881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.28@tcp -d 192.168.201.128@tcp -m GET -e network_timeout -r 1 [ 2924.358391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.28@tcp -d 192.168.201.28@tcp -m GET -e network_timeout -r 1 [ 2924.394684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.128@tcp [ 2924.412417] LNet: There was an unexpected network error while writing to 192.168.201.128: rc = -22 [ 2924.643835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2924.681804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2924.703419] LNet: 1 local NIs in recovery (showing 1): 192.168.201.28@tcp [ 2924.725332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2924.759652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2924.791392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2924.823169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2924.872802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2924.941478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2924.982862] LNet: 192211:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2925.015943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2925.084760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2925.134959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2925.197717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2925.235089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2925.295702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2925.760132] LNet: 192250:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2925.771603] LNetError: 192250:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2925.796979] LNet: Removed LNI 192.168.201.28@tcp [ 2926.414603] Key type .llcrypt unregistered [ 2926.421527] Key type ._llcrypt unregistered [ 2952.498985] Key type ._llcrypt registered [ 2952.503388] Key type .llcrypt registered [ 2952.860562] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2952.876681] alg: No test for adler32 (adler32-zlib) [ 2953.874255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2953.908191] LNet: Added LNI 192.168.201.28@tcp [8/256/0/180] [ 2964.054231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.128@tcp [ 2978.246104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2978.272749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2978.312635] LNet: Added LNI 192.168.201.28@tcp1 [8/256/0/180] [ 2980.979732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.28@tcp -d 192.168.201.128@tcp -m GET -e network_timeout -r 1 [ 2981.051325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.28@tcp -d 192.168.201.128@tcp1 -m GET -e network_timeout -r 1 [ 2981.130856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.28@tcp -d 192.168.201.28@tcp -m GET -e network_timeout -r 1 [ 2981.191703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.28@tcp -d 192.168.201.28@tcp1 -m GET -e network_timeout -r 1 [ 2981.251503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.28@tcp1 -d 192.168.201.128@tcp -m GET -e network_timeout -r 1 [ 2981.316957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.28@tcp1 -d 192.168.201.128@tcp1 -m GET -e network_timeout -r 1 [ 2981.378442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.28@tcp1 -d 192.168.201.28@tcp -m GET -e network_timeout -r 1 [ 2981.441891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.28@tcp1 -d 192.168.201.28@tcp1 -m GET -e network_timeout -r 1 [ 2981.522914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.128@tcp [ 2981.538354] LNet: There was an unexpected network error while writing to 192.168.201.128: rc = -22 [ 2981.599736] LNet: 1 local NIs in recovery (showing 1): 192.168.201.28@tcp [ 2981.607883] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.128@tcp [ 2981.799983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2981.860067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2981.913036] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2981.965071] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2982.015400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2982.084949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2982.129981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2982.175466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2982.196348] LNet: 193132:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2982.223561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2982.271134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2982.325917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2982.371575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2982.437646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2982.503942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2987.824616] LNet: There was an unexpected network error while writing to 192.168.201.128: rc = -22 [ 2988.454408] LNet: 193206:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2988.466343] LNetError: 193206:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2988.479311] LNet: Removed LNI 192.168.201.28@tcp [ 2988.515645] LNet: Removed LNI 192.168.201.28@tcp1 [ 2989.056743] Key type .llcrypt unregistered [ 2989.061474] Key type ._llcrypt unregistered [ 3010.141630] Key type ._llcrypt registered [ 3010.144500] Key type .llcrypt registered [ 3011.833106] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 11:45:47 (1788191147) [ 3012.476391] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3012.510086] alg: No test for adler32 (adler32-zlib) [ 3013.466556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3013.561676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 3013.605626] LNet: Added LNI 192.168.201.28@tcp [8/256/0/180] [ 3014.783253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 3014.810667] LNet: Added LNI 192.168.201.28@tcp1 [8/256/0/180] [ 3016.481533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.28@tcp [ 3016.568752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 3016.627836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 3016.679737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 3016.709838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 3016.738854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 3016.772783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.201.28@tcp [ 3016.791101] LNetError: 194313:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.201.28@tcp to recovery [ 3017.631409] LNet: 1 local NIs in recovery (showing 1): 192.168.201.28@tcp [ 3017.631676] LNet: There was an unexpected network error while writing to 192.168.201.28: rc = -22 [ 3019.683023] LNet: There was an unexpected network error while writing to 192.168.201.28: rc = -22 [ 3020.691813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 3023.775487] LNet: There was an unexpected network error while writing to 192.168.201.28: rc = -22 [ 3024.572294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 3024.637757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 3024.743050] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3024.795961] LNet: 194420:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3024.816675] LNetError: 194420:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3024.826699] LNet: Removed LNI 192.168.201.28@tcp [ 3024.847126] LNet: Removed LNI 192.168.201.28@tcp1 [ 3024.906578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3025.002712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 3025.041235] LNet: Added LNI 192.168.201.28@tcp [8/256/0/180] [ 3025.965607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 3025.999611] LNet: Added LNI 192.168.201.28@tcp1 [8/256/0/180] [ 3027.176647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.28@tcp [ 3027.266559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 3027.320945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 3027.357323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 3027.407814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 3027.443299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 3027.484501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 3027.519285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.201.28@tcp [ 3027.593834] LNetError: 194693:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.201.28@tcp to recovery [ 3028.063426] LNet: There was an unexpected network error while writing to 192.168.201.28: rc = -22 [ 3031.347201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 3034.207710] LNet: There was an unexpected network error while writing to 192.168.201.28: rc = -22 [ 3034.209826] LNet: Skipped 1 previous similar message [ 3038.522470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 3038.542827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 3038.569462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 3047.546708] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 11:46:23 (1788191183) [ 3047.635641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3047.666444] LNet: 195401:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3047.673288] LNetError: 195401:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3047.678684] LNetError: 195401:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 3047.684550] LNet: Removed LNI 192.168.201.28@tcp [ 3047.695633] LNet: Removed LNI 192.168.201.28@tcp1 [ 3047.719960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3047.786853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 3047.802610] LNet: Added LNI 192.168.201.28@tcp [8/256/0/180] [ 3048.421515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 3048.445309] LNet: Added LNI 192.168.201.28@tcp1 [8/256/0/180] [ 3049.306735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.28@tcp [ 3049.432339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 3049.488532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 3049.526390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 3049.564922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 3049.599910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 3049.638587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.201.28@tcp --health 0 [ 3049.781510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 3049.823321] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.28@tcp [ 3050.848719] LNet: There was an unexpected network error while writing to 192.168.201.28: rc = -22 [ 3050.859927] LNet: Skipped 1 previous similar message [ 3064.701763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 3065.003155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 3065.096315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 3065.169273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.201.28@tcp --health 500 [ 3065.297988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 3072.084570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 3072.157329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3072.203840] LNet: 195893:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3072.217502] LNetError: 195893:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3072.226747] LNetError: 195893:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 3072.237880] LNet: Removed LNI 192.168.201.28@tcp [ 3072.255213] LNet: Removed LNI 192.168.201.28@tcp1 [ 3072.275957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3072.317239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 3072.337789] LNet: Added LNI 192.168.201.28@tcp [8/256/0/180] [ 3073.154914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 3073.186372] LNet: Added LNI 192.168.201.28@tcp1 [8/256/0/180] [ 3074.453701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.28@tcp [ 3074.487690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 3074.543573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 3074.574197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 3074.603716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 3074.637393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 3074.666218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 3074.715754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.201.28@tcp --health 0 [ 3075.359397] LNet: There was an unexpected network error while writing to 192.168.201.28: rc = -22 [ 3075.370136] LNet: Skipped 3 previous similar messages [ 3076.120267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 3086.811979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 3086.856166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 3086.901604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 3095.424604] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 11:47:11 (1788191231) [ 3096.407558] LNet: 196949:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3096.416829] LNetError: 196949:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3096.426319] LNetError: 196949:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 3096.432112] LNet: Removed LNI 192.168.201.28@tcp [ 3096.442746] LNet: Removed LNI 192.168.201.28@tcp1 [ 3097.352056] Key type .llcrypt unregistered [ 3097.356757] Key type ._llcrypt unregistered [ 3104.435355] Key type ._llcrypt registered [ 3104.439021] Key type .llcrypt registered [ 3104.790878] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 3104.812256] alg: No test for adler32 (adler32-zlib) [ 3105.794716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 3105.843729] LNet: Added LNI 192.168.201.28@tcp [8/256/0/180] [ 3115.742867] Lustre: DEBUG MARKER: Initial discovery [ 3115.788107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.201.128@tcp [ 3118.603305] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 3118.656480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 3118.717995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.201.128@tcp [ 3118.758700] LNet: There was an unexpected network error while writing to 192.168.201.128: rc = -22 [ 3118.792291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 3118.854300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.201.128@tcp -r 1 -m GET -l 3 [ 3118.905713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.201.128@tcp -r 1 -m GET -e local_error [ 3118.953767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.201.128@tcp -r 1 -m PUT -l 6 [ 3119.136884] LNet: 1 local NIs in recovery (showing 1): 192.168.201.28@tcp [ 3120.393261] Lustre: DEBUG MARKER: Force oleg128-client.virtnet to discover 192.168.201.128@tcp (in background) [ 3120.793714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.201.128@tcp [ 3122.825928] Lustre: DEBUG MARKER: Force oleg128-server to discover 192.168.201.28@tcp [ 3123.882787] LNet: There was an unexpected network error while writing to 192.168.201.128: rc = -22 [ 3125.574275] Lustre: DEBUG MARKER: Wait for 197717 [ 3127.400313] LNet: There was an unexpected network error while writing to 192.168.201.128: rc = -22 [ 3127.494350] Lustre: DEBUG MARKER: Finished wait on 197717 [ 3127.639690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay del -a [ 3127.688292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 3128.055699] LNet: 198171:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3128.066909] LNetError: 198171:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3128.092362] LNet: Removed LNI 192.168.201.28@tcp [ 3128.533568] Key type .llcrypt unregistered [ 3128.538514] Key type ._llcrypt unregistered [ 3146.274110] Key type ._llcrypt registered [ 3146.276268] Key type .llcrypt registered [ 3148.243914] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 11:48:03 (1788191283) [ 3148.738379] Key type .llcrypt unregistered [ 3148.741442] Key type ._llcrypt unregistered [ 3154.896639] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 3155.108189] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 3155.111745] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 3155.114831] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 3157.707619] Key type ._llcrypt registered [ 3157.710826] Key type .llcrypt registered [ 3157.926455] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 3157.951636] alg: No test for adler32 (adler32-zlib) [ 3158.998357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3159.064321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 3159.082607] LNet: Added LNI 192.168.201.28@tcp [8/256/0/180] [ 3159.961311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 3159.995384] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 3168.907314] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 11:48:24 (1788191304) [ 3169.089511] LNet: 200264:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3169.100728] LNetError: 200264:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 3169.111481] LNet: Removed LNI 10.1.2.3@tcp [ 3169.136353] LNet: Removed LNI 192.168.201.28@tcp [ 3169.721061] Key type .llcrypt unregistered [ 3169.724583] Key type ._llcrypt unregistered [ 3177.085395] Key type ._llcrypt registered [ 3177.097370] Key type .llcrypt registered [ 3177.419652] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 3177.449977] alg: No test for adler32 (adler32-zlib) [ 3178.437410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3178.511499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 3178.538411] LNet: Added LNI 192.168.201.28@tcp [8/256/0/180] [ 3179.485580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 3179.522187] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 3180.821841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.201.28@tcp [ 3181.921171] LNetError: Unexpected error -22 connecting to 192.168.201.28@tcp at host 192.168.201.28:988 [ 3188.779705] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 11:48:44 (1788191324) [ 3189.259847] LNet: 201406:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3189.276591] LNetError: 201406:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 3189.287275] LNet: Removed LNI 10.1.2.3@tcp [ 3189.318776] LNet: Removed LNI 192.168.201.28@tcp [ 3190.273905] Key type .llcrypt unregistered [ 3190.279824] Key type ._llcrypt unregistered [ 3202.159335] Key type ._llcrypt registered [ 3202.162928] Key type .llcrypt registered [ 3202.447178] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 3202.466518] alg: No test for adler32 (adler32-zlib) [ 3203.498466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3203.579616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 3203.627047] LNet: Added LNI 192.168.201.28@tcp1 [8/256/0/180] [ 3204.503152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 3204.519590] LNet: Added LNI 192.168.201.28@tcp2 [8/256/0/180] [ 3205.531534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.201.28@tcp1 --nid 192.168.201.28@tcp2 [ 3212.952302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.28@tcp1 192.168.201.28@tcp2 [ 3212.986519] LNetError: 202241:0:(lib-move.c:2101:lnet_handle_find_routed_path()) Unable to send message from 192.168.201.28@tcp1 to 192.168.201.28@tcp2 - Route table may be misconfigured [ 3213.013518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.28@tcp2 192.168.201.28@tcp1 [ 3222.808859] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 11:49:18 (1788191358) [ 3223.171685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3223.269380] LNet: 202823:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3223.276619] LNetError: 202823:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3223.288869] LNet: Removed LNI 192.168.201.28@tcp1 [ 3224.358223] LNet: Removed LNI 192.168.201.28@tcp2 [ 3224.446625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3224.549291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 3224.601435] LNet: Added LNI 192.168.201.28@tcp [8/256/0/180] [ 3225.727336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 3225.767217] LNet: Added LNI 192.168.201.28@tcp1 [8/256/0/180] [ 3227.232807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.28@tcp [ 3227.336851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.28@tcp 192.168.201.28@tcp [ 3227.411232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.28@tcp1 192.168.201.28@tcp1 [ 3227.505116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.201.28@tcp -d 192.168.201.28@tcp -e network_timeout [ 3227.592822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.201.28@tcp -d 192.168.201.28@tcp1 -e network_timeout [ 3227.679678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.201.28@tcp1 -d 192.168.201.28@tcp -e network_timeout [ 3227.791041] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.201.28@tcp1 -d 192.168.201.28@tcp1 -e network_timeout [ 3227.894178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.28@tcp [ 3227.934759] LNet: There was an unexpected network error while writing to 192.168.201.28: rc = -22 [ 3227.976506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 3228.117586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 3228.288302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 3228.575282] LNet: 1 local NIs in recovery (showing 1): 192.168.201.28@tcp [ 3236.473083] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 11:49:31 (1788191371) [ 3236.734470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3236.756920] LNet: 203692:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3236.763685] LNetError: 203692:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3236.768536] LNetError: 203692:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 3236.780433] LNet: Removed LNI 192.168.201.28@tcp [ 3237.812360] LNet: Removed LNI 192.168.201.28@tcp1 [ 3237.841236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3237.914750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 3238.123922] LNet: 203731:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3238.597606] Key type .llcrypt unregistered [ 3238.604917] Key type ._llcrypt unregistered [ 3250.977794] Key type ._llcrypt registered [ 3250.983687] Key type .llcrypt registered [ 3252.989162] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 11:49:48 (1788191388) [ 3253.272219] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3253.293855] alg: No test for adler32 (adler32-zlib) [ 3254.432171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3256.719514] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 3258.867499] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 11:49:54 (1788191394) [ 3259.003889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3259.042291] LNet: 204835:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3259.098650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3259.223604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 3259.271983] LNet: Added LNI 192.168.201.28@tcp [8/256/0/180] [ 3260.338898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 3260.388726] LNet: Added LNI 192.168.201.28@tcp1 [8/256/0/180] [ 3262.151920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.28@tcp [ 3262.227254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.28@tcp1 [ 3262.309241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.28@tcp1 [ 3272.737842] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 11:50:08 (1788191408) [ 3274.880414] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg128-server [ 3277.599524] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 11:50:13 (1788191413) [ 3280.173810] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg128-server [ 3282.944043] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 11:50:18 (1788191418) [ 3284.964841] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg128-server [ 3287.830537] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 11:50:23 (1788191423) [ 3290.871096] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg128-server [ 3294.017970] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 11:50:29 (1788191429) [ 3296.926869] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg128-server [ 3299.523994] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 11:50:34 (1788191434) [ 3302.077659] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg128-server [ 3304.593419] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 11:50:39 (1788191439) [ 3307.004662] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg128-server [ 3309.289190] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 11:50:45 (1788191445) [ 3311.798232] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg128-server [ 3313.949461] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 11:50:49 (1788191449) [ 3315.951852] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg128-server [ 3317.979754] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 11:50:53 (1788191453) [ 3318.398608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3318.426512] LNet: 208783:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3318.435236] LNetError: 208783:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3318.449351] LNet: Removed LNI 192.168.201.28@tcp [ 3319.529776] LNet: Removed LNI 192.168.201.28@tcp1 [ 3319.582208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3319.756877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 3319.822574] LNet: Added LNI 192.168.201.28@tcp [8/256/0/180] [ 3320.802810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 3320.925169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.28@tcp [ 3321.173365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3321.224528] LNet: 208931:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3321.255232] LNetError: 208931:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3321.268953] LNetError: 208931:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 3321.300689] LNet: Removed LNI 192.168.201.28@tcp [ 3321.326586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3321.433898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 3321.476422] LNet: Added LNI 192.168.201.28@tcp [8/256/0/180] [ 3322.826276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.28@tcp [ 3323.071935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3323.126889] LNetError: 209076:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3323.147061] LNet: Removed LNI 192.168.201.28@tcp [ 3323.205192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3323.309872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 3323.372438] LNet: Added LNI 192.168.201.28@tcp [8/256/0/180] [ 3324.039915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 3324.121098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.28@tcp [ 3324.485793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3324.571518] LNet: 209223:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3324.588958] LNet: 209223:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 3325.609377] LNet: Removed LNI 192.168.201.28@tcp [ 3325.680498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3325.780904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 3325.826844] LNet: Added LNI 192.168.201.28@tcp [8/256/0/180] [ 3326.825527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.28@tcp [ 3326.981982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3327.005961] LNetError: 209369:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3327.010186] LNetError: 209369:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 3328.046125] LNet: Removed LNI 192.168.201.28@tcp [ 3328.098196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3328.172970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 3328.195893] LNet: Added LNI 192.168.201.28@tcp [8/256/0/180] [ 3329.220880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 3329.359967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.28@tcp [ 3329.700937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3329.720732] LNet: 209516:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3329.727865] LNet: 209516:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 3330.793219] LNet: Removed LNI 192.168.201.28@tcp [ 3330.877661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3331.019692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 3331.072614] LNet: Added LNI 192.168.201.28@tcp [8/256/0/180] [ 3332.274302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.28@tcp [ 3332.551879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3332.583995] LNetError: 209661:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3332.590217] LNetError: 209661:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 3333.613768] LNet: Removed LNI 192.168.201.28@tcp [ 3333.641221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3333.679685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 3333.698210] LNet: Added LNI 192.168.201.28@tcp [8/256/0/180] [ 3334.278349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 3334.341135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.28@tcp [ 3335.927560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3337.002317] LNet: Removed LNI 192.168.201.28@tcp [ 3337.045104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3337.179488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 3337.267732] LNet: Added LNI 192.168.201.28@tcp [8/256/0/180] [ 3338.851045] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.28@tcp [ 3339.214046] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3339.254878] LNet: 209964:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3339.268468] LNet: 209964:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 3340.328675] LNet: Removed LNI 192.168.201.28@tcp [ 3340.422835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3340.541615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 3340.618661] LNet: Added LNI 192.168.201.28@tcp [8/256/0/180] [ 3341.882773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 3341.959104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.28@tcp [ 3342.158905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3342.181875] LNetError: 210111:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3342.191252] LNetError: 210111:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 3342.208345] LNet: Removed LNI 192.168.201.28@tcp [ 3342.239345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3342.334902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 3342.370266] LNet: Added LNI 192.168.201.28@tcp [8/256/0/180] [ 3343.700501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.28@tcp [ 3346.312641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3347.366730] LNet: Removed LNI 192.168.201.28@tcp [ 3347.441367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3347.599641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 3347.665447] LNet: Added LNI 192.168.201.28@tcp [8/256/0/180] [ 3349.014833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 3349.152347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.28@tcp [ 3349.514518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3350.565803] LNet: Removed LNI 192.168.201.28@tcp [ 3350.607236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3350.744402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 3350.800697] LNet: Added LNI 192.168.201.28@tcp [8/256/0/180] [ 3352.364650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.28@tcp [ 3354.055431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3355.180718] LNet: Removed LNI 192.168.201.28@tcp [ 3355.252563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3355.374236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 3355.430535] LNet: Added LNI 192.168.201.28@tcp [8/256/0/180] [ 3356.029984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 3356.086573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3356.101398] LNet: 210710:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3356.104084] LNet: 210710:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 3356.118376] LNet: Removed LNI 192.168.201.28@tcp [ 3356.133546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3356.183527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 3356.211608] LNet: Added LNI 192.168.201.28@tcp [8/256/0/180] [ 3357.676303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 3357.800698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3357.840379] LNet: Removed LNI 192.168.201.28@tcp [ 3357.870066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3357.967822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 128 [ 3358.092427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3358.193615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3358.304365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 1000000000000000000000000 [ 3358.411464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3358.466922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3358.539568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer foo [ 3358.650923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3358.707370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3358.782280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 8 [ 3358.826912] LNet: Added LNI 192.168.201.28@tcp [8/256/0/180] [ 3360.048849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3360.070478] LNetError: 211018:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3360.075695] LNetError: 211018:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 3360.086537] LNet: Removed LNI 192.168.201.28@tcp [ 3360.099661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3360.162447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 3368.312963] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 11:51:44 (1788191504) [ 3368.406892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3368.450568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3368.522843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 3368.545295] LNet: Added LNI 192.168.201.28@tcp231 [8/256/0/180] [ 3369.479253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3369.504150] LNet: Removed LNI 192.168.201.28@tcp231 [ 3369.517040] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3369.575485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 3369.604734] LNet: Added LNI 192.168.201.28@tcp231 [8/256/0/180] [ 3369.609931] LNet: 211754:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 3370.243885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 3370.297889] LNet: Removed LNI 192.168.201.28@tcp231 [ 3370.319170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 3370.368412] LNet: Added LNI 192.168.201.28@tcp231 [8/256/0/180] [ 3371.963272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3372.010803] LNet: Removed LNI 192.168.201.28@tcp231 [ 3372.041635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3372.155113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 3372.205268] LNet: Added LNI 192.168.201.28@tcp231 [8/256/0/0] [ 3372.222690] LNet: 211906:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 3372.811575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 3372.851669] LNet: Removed LNI 192.168.201.28@tcp231 [ 3372.870812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 3372.908568] LNet: Added LNI 192.168.201.28@tcp231 [8/256/0/0] [ 3374.565729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3374.595382] LNet: Removed LNI 192.168.201.28@tcp231 [ 3374.621608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3374.673671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 3374.694783] LNet: Added LNI 192.168.201.28@tcp231 [8/256/0/60] [ 3374.698672] LNet: 212058:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 3375.082590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 3375.121355] LNet: Removed LNI 192.168.201.28@tcp231 [ 3375.146543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 3375.189069] LNet: Added LNI 192.168.201.28@tcp231 [8/256/0/60] [ 3376.490458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3376.531207] LNet: Removed LNI 192.168.201.28@tcp231 [ 3376.581647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3376.736070] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 3376.811580] LNet: Added LNI 192.168.201.28@tcp231 [8/256/0/180] [ 3376.817102] LNet: 212209:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 3377.428092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 3377.465812] LNet: Removed LNI 192.168.201.28@tcp231 [ 3377.489466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 3377.523151] LNet: Added LNI 192.168.201.28@tcp231 [8/256/0/180] [ 3388.320582] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 11:52:03 (1788191523) [ 3388.593801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3388.671737] LNet: 212927:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3388.695043] LNet: 212927:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 10 previous similar messages [ 3388.743662] LNet: Removed LNI 192.168.201.28@tcp231 [ 3388.782956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3388.907524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 3388.929149] LNet: Added LNI 192.168.201.28@tcp232 [8/256/0/180] [ 3389.656953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 3389.720981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 3389.772824] LNet: Removed LNI 192.168.201.28@tcp232 [ 3389.790788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 3389.817742] LNet: Added LNI 192.168.201.28@tcp232 [8/256/0/180] [ 3390.697964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 3390.724371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 3390.753376] LNet: Removed LNI 192.168.201.28@tcp232 [ 3390.775451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 3390.799619] LNet: Added LNI 192.168.201.28@tcp232 [8/256/0/180] [ 3391.646839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 3391.697378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 3391.726703] LNet: Removed LNI 192.168.201.28@tcp232 [ 3391.737816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 3391.775752] LNet: Added LNI 192.168.201.28@tcp232 [8/256/0/180] [ 3392.542180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 3392.580798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 3392.611824] LNet: Removed LNI 192.168.201.28@tcp232 [ 3392.628717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 3401.404496] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 11:52:17 (1788191537) [ 3403.503695] Key type .llcrypt unregistered [ 3403.508257] Key type ._llcrypt unregistered [ 3412.390878] Key type ._llcrypt registered [ 3412.394818] Key type .llcrypt registered [ 3412.692740] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 3412.703555] alg: No test for adler32 (adler32-zlib) [ 3413.556307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 3413.572608] LNet: Added LNI 192.168.201.28@tcp [8/256/0/180] [ 3422.842437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.128@tcp [ 3433.531744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3433.583505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 3433.610210] LNet: Added LNI 192.168.201.28@tcp1 [8/256/0/180] [ 3435.055802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.28@tcp -d 192.168.201.128@tcp -m GET -e local_interrupt -r 2 [ 3435.084547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.28@tcp -d 192.168.201.128@tcp1 -m GET -e local_interrupt -r 2 [ 3435.118955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.28@tcp -d 192.168.201.28@tcp -m GET -e local_interrupt -r 2 [ 3435.152608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.28@tcp -d 192.168.201.28@tcp1 -m GET -e local_interrupt -r 2 [ 3435.191717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.28@tcp1 -d 192.168.201.128@tcp -m GET -e local_interrupt -r 2 [ 3435.245911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.28@tcp1 -d 192.168.201.128@tcp1 -m GET -e local_interrupt -r 2 [ 3435.289734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.28@tcp1 -d 192.168.201.28@tcp -m GET -e local_interrupt -r 2 [ 3435.346792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.28@tcp1 -d 192.168.201.28@tcp1 -m GET -e local_interrupt -r 2 [ 3435.384341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.28@tcp 192.168.201.128@tcp [ 3435.409224] LNet: There was an unexpected network error while writing to 192.168.201.128: rc = -22 [ 3435.416839] LNet: 1 local NIs in recovery (showing 1): 192.168.201.28@tcp [ 3435.791658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 3435.832863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 3435.862458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 3435.937515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 3435.986913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 3436.010775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 3436.050696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 3436.098874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 3436.147336] LNet: 214730:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 3436.197389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 3436.238367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 3436.275556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 3436.296667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 3436.315680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 3436.342515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 3437.072280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.28@tcp -d 192.168.201.128@tcp -m GET -e local_dropped -r 2 [ 3437.131828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.28@tcp -d 192.168.201.128@tcp1 -m GET -e local_dropped -r 2 [ 3437.202981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.28@tcp -d 192.168.201.28@tcp -m GET -e local_dropped -r 2 [ 3437.273604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.28@tcp -d 192.168.201.28@tcp1 -m GET -e local_dropped -r 2 [ 3437.355383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.28@tcp1 -d 192.168.201.128@tcp -m GET -e local_dropped -r 2 [ 3437.399621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.28@tcp1 -d 192.168.201.128@tcp1 -m GET -e local_dropped -r 2 [ 3437.455685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.28@tcp1 -d 192.168.201.28@tcp -m GET -e local_dropped -r 2 [ 3437.501830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.28@tcp1 -d 192.168.201.28@tcp1 -m GET -e local_dropped -r 2 [ 3437.549529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.28@tcp 192.168.201.128@tcp [ 3437.580157] LNet: There was an unexpected network error while writing to 192.168.201.128: rc = -22 [ 3437.868741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 3437.907523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 3437.953251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 3438.023258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 3438.070640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 3438.101114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 3438.129957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 3438.155443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 3438.172363] LNet: 214851:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 3438.186360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 3438.214403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 3438.247759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 3438.278842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 3438.304610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 3438.337961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 3438.978409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.28@tcp -d 192.168.201.128@tcp -m GET -e local_aborted -r 2 [ 3439.051286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.28@tcp -d 192.168.201.128@tcp1 -m GET -e local_aborted -r 2 [ 3439.164681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.28@tcp -d 192.168.201.28@tcp -m GET -e local_aborted -r 2 [ 3439.261679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.28@tcp -d 192.168.201.28@tcp1 -m GET -e local_aborted -r 2 [ 3439.323322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.28@tcp1 -d 192.168.201.128@tcp -m GET -e local_aborted -r 2 [ 3439.386584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.28@tcp1 -d 192.168.201.128@tcp1 -m GET -e local_aborted -r 2 [ 3439.436729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.28@tcp1 -d 192.168.201.28@tcp -m GET -e local_aborted -r 2 [ 3439.503755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.28@tcp1 -d 192.168.201.28@tcp1 -m GET -e local_aborted -r 2 [ 3439.574993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.28@tcp 192.168.201.128@tcp [ 3439.622704] LNet: There was an unexpected network error while writing to 192.168.201.128: rc = -22 [ 3439.873889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 3439.919687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 3439.946641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 3439.976521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 3440.004783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 3440.034585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 3440.080425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 3440.126601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 3440.168458] LNet: 214971:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 3440.206468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 3440.248993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 3440.287677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 3440.327631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 3440.363526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 3440.403779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 3441.164181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.28@tcp -d 192.168.201.128@tcp -m GET -e local_no_route -r 2 [ 3441.199929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.28@tcp -d 192.168.201.128@tcp1 -m GET -e local_no_route -r 2 [ 3441.231637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.28@tcp -d 192.168.201.28@tcp -m GET -e local_no_route -r 2 [ 3441.294603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.28@tcp -d 192.168.201.28@tcp1 -m GET -e local_no_route -r 2 [ 3441.366783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.28@tcp1 -d 192.168.201.128@tcp -m GET -e local_no_route -r 2 [ 3441.444756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.28@tcp1 -d 192.168.201.128@tcp1 -m GET -e local_no_route -r 2 [ 3441.518411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.28@tcp1 -d 192.168.201.28@tcp -m GET -e local_no_route -r 2 [ 3441.561347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.28@tcp1 -d 192.168.201.28@tcp1 -m GET -e local_no_route -r 2 [ 3441.620690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.28@tcp 192.168.201.128@tcp [ 3441.677064] LNet: There was an unexpected network error while writing to 192.168.201.128: rc = -22 [ 3441.998581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 3442.022581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 3442.046824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 3442.073076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 3442.102472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 3442.151994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 3442.177439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 3442.211779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 3442.227897] LNet: 215092:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 3442.247862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 3442.278904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 3442.307883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 3442.336891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 3442.365996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 3442.410433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 3443.000403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.28@tcp -d 192.168.201.128@tcp -m GET -e local_timeout -r 2 [ 3443.065571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.28@tcp -d 192.168.201.128@tcp1 -m GET -e local_timeout -r 2 [ 3443.114584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.28@tcp -d 192.168.201.28@tcp -m GET -e local_timeout -r 2 [ 3443.157637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.28@tcp -d 192.168.201.28@tcp1 -m GET -e local_timeout -r 2 [ 3443.209695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.28@tcp1 -d 192.168.201.128@tcp -m GET -e local_timeout -r 2 [ 3443.261819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.28@tcp1 -d 192.168.201.128@tcp1 -m GET -e local_timeout -r 2 [ 3443.326876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.28@tcp1 -d 192.168.201.28@tcp -m GET -e local_timeout -r 2 [ 3443.365977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.28@tcp1 -d 192.168.201.28@tcp1 -m GET -e local_timeout -r 2 [ 3443.416711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.28@tcp 192.168.201.128@tcp [ 3443.742363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 3443.767686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 3443.794946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 3443.823413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 3443.845821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 3443.869077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 3443.917779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 3443.977528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 3444.005506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 3444.064666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 3444.121722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 3444.177720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 3444.237645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 3444.286267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 3445.061282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.28@tcp -d 192.168.201.128@tcp -m GET -e remote_dropped -r 2 [ 3445.103570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.28@tcp -d 192.168.201.128@tcp1 -m GET -e remote_dropped -r 2 [ 3445.147684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.28@tcp -d 192.168.201.28@tcp -m GET -e remote_dropped -r 2 [ 3445.197840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.28@tcp -d 192.168.201.28@tcp1 -m GET -e remote_dropped -r 2 [ 3445.241544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.28@tcp1 -d 192.168.201.128@tcp -m GET -e remote_dropped -r 2 [ 3445.282972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.28@tcp1 -d 192.168.201.128@tcp1 -m GET -e remote_dropped -r 2 [ 3445.329428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.28@tcp1 -d 192.168.201.28@tcp -m GET -e remote_dropped -r 2 [ 3445.385757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.28@tcp1 -d 192.168.201.28@tcp1 -m GET -e remote_dropped -r 2 [ 3445.432458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.28@tcp 192.168.201.128@tcp [ 3445.459063] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.128@tcp [ 3445.782881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 3445.832730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 3445.873959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 3445.911678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 3445.948674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 3445.992475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 3446.034367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 3446.078500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 3446.146658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 3446.188308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 3446.233638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 3446.273900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 3446.319806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 3446.359787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 3452.366432] LNet: 215406:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3452.396352] LNetError: 215406:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3452.405961] LNet: Removed LNI 192.168.201.28@tcp [ 3452.419944] LNet: Removed LNI 192.168.201.28@tcp1 [ 3452.881607] Key type .llcrypt unregistered [ 3452.883165] Key type ._llcrypt unregistered [ 3474.615243] Key type ._llcrypt registered [ 3474.623970] Key type .llcrypt registered [ 3476.032237] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 11:53:31 (1788191611) [ 3476.400060] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3476.413365] alg: No test for adler32 (adler32-zlib) [ 3477.368793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3477.436609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 3477.467591] LNet: Added LNI 192.168.201.28@tcp [8/256/0/180] [ 3478.234883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 3478.262342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.201.28@tcp [ 3478.289345] LNetError: 216395:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.201.28@tcp to recovery [ 3478.379270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.201.28@tcp [ 3478.484406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 3485.560971] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 11:53:41 (1788191621) [ 3486.099926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3486.133851] LNet: 216994:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3486.140135] LNetError: 216994:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3486.168681] LNet: Removed LNI 192.168.201.28@tcp [ 3486.212714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3486.261933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 3486.292299] LNet: Added LNI 192.168.201.28@tcp [8/256/0/180] [ 3487.138770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 3487.156279] LNet: Added LNI 192.168.201.28@tcp1 [8/256/0/180] [ 3488.150931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.201.28@tcp [ 3488.165150] LNetError: 217260:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.201.28@tcp to recovery [ 3488.287357] LNet: 1 local NIs in recovery (showing 1): 192.168.201.28@tcp [ 3488.290731] LNet: There was an unexpected network error while writing to 192.168.201.28: rc = -22 [ 3489.412863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.201.28@tcp [ 3496.706645] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 11:53:52 (1788191632) [ 3498.809561] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg128-server [ 3500.556233] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 11:53:56 (1788191636) [ 3502.587547] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg128-server [ 3504.869968] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 11:54:00 (1788191640) [ 3507.141358] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg128-server [ 3508.725302] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 11:54:05 (1788191645) [ 3509.173823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3509.201242] LNet: 218909:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3509.207547] LNetError: 218909:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3509.214327] LNet: Removed LNI 192.168.201.28@tcp [ 3509.225309] LNet: Removed LNI 192.168.201.28@tcp1 [ 3509.256832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3509.304500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 3509.332369] LNet: Added LNI 192.168.201.28@tcp [8/256/0/180] [ 3510.267759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 3510.354650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 3510.392701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 3510.479420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 3510.513365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 3510.574285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 3517.395409] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 11:54:13 (1788191653) [ 3517.613239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3517.630212] LNet: 219649:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3517.633896] LNetError: 219649:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3517.636910] LNetError: 219649:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 3517.650259] LNet: Removed LNI 192.168.201.28@tcp [ 3517.685869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3517.793552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 3517.846647] LNet: Added LNI 192.168.201.28@tcp [8/256/0/180] [ 3526.927483] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 11:54:22 (1788191662) [ 3526.991783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3527.021278] LNet: 220354:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3527.028170] LNetError: 220354:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3527.079267] LNet: Removed LNI 192.168.201.28@tcp [ 3527.119315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3527.307209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 3527.349918] LNet: Added LNI 192.168.201.28@tcp [8/256/0/180] [ 3537.235988] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 11:54:32 (1788191672) [ 3539.440912] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 3541.949575] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 11:54:37 (1788191677) [ 3543.164793] LNet: 221410:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3543.175767] LNetError: 221410:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3543.191272] LNet: Removed LNI 192.168.201.28@tcp [ 3544.119355] Key type .llcrypt unregistered [ 3544.122935] Key type ._llcrypt unregistered [ 3553.080892] Key type ._llcrypt registered [ 3553.083316] Key type .llcrypt registered [ 3553.292368] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 3553.309495] alg: No test for adler32 (adler32-zlib) [ 3554.264308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 3554.294768] LNet: Added LNI 192.168.201.28@tcp [8/256/0/180] [ 3565.846450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.128@tcp [ 3574.719367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.201.128@tcp [ 3575.396661] LNet: 221828:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3575.414342] LNetError: 221828:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3575.455487] LNet: Removed LNI 192.168.201.28@tcp [ 3576.650627] Key type .llcrypt unregistered [ 3576.664722] Key type ._llcrypt unregistered [ 3592.811606] Key type ._llcrypt registered [ 3592.817723] Key type .llcrypt registered [ 3594.709824] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 3596.582258] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 3598.817513] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 11:55:34 (1788191734) [ 3600.891513] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 [ 3602.677600] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 11:55:38 (1788191738) [ 3604.743667] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg128-server [ 3606.462371] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 11:55:42 (1788191742) [ 3608.370707] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg128-server [ 3611.314547] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 11:55:46 (1788191746) [ 3613.331787] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3613.579122] alg: No test for adler32 (adler32-zlib) [ 3614.709806] Lustre: Lustre: Build Version: 2.17.57_85_gb71fc30 [ 3614.988628] LNet: Added LNI 192.168.201.28@tcp [8/256/0/180] [ 3616.679828] Key type lgssc registered [ 3618.216979] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3643.247094] Key type lgssc unregistered [ 3643.624761] LNet: 225070:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3643.635698] LNetError: 225070:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3643.660772] LNet: Removed LNI 192.168.201.28@tcp [ 3644.699102] Key type .llcrypt unregistered [ 3644.701614] Key type ._llcrypt unregistered [ 3662.245481] Key type ._llcrypt registered [ 3662.247314] Key type .llcrypt registered [ 3663.563517] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 11:56:39 (1788191799) [ 3664.059096] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3664.078908] alg: No test for adler32 (adler32-zlib) [ 3665.055890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3665.145676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 3665.203448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 3665.270883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 3665.341705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 3672.212590] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 11:56:48 (1788191808) [ 3672.463702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3672.501623] LNet: 226497:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3672.557454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3672.620777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 3672.648346] LNet: Added LNI 192.168.201.28@tcp [8/256/0/180] [ 3673.655940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3673.685985] LNet: 226643:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3673.689798] LNetError: 226643:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3673.708111] LNet: Removed LNI 192.168.201.28@tcp [ 3673.726893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3673.787886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 3673.814154] LNet: Added LNI 192.168.201.28@tcp [8/256/0/180] [ 3674.730672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3674.747261] LNet: 226671:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3674.753327] LNetError: 226671:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3674.768530] LNet: Removed LNI 192.168.201.28@tcp [ 3674.801374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3674.878992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 3674.900572] LNet: Added LNI 192.168.201.28@tcp [8/256/0/180] [ 3682.493557] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 11:56:58 (1788191818) [ 3682.741347] LNet: 227268:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3682.749944] LNetError: 227268:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3682.783441] LNet: Removed LNI 192.168.201.28@tcp [ 3683.551704] Key type .llcrypt unregistered [ 3683.557883] Key type ._llcrypt unregistered [ 3691.403413] Key type ._llcrypt registered [ 3691.406334] Key type .llcrypt registered [ 3691.596967] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 3691.613294] alg: No test for adler32 (adler32-zlib) [ 3692.607825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3692.702837] LNet: Added LNI 192.168.201.28@tcp [8/256/0/180] [ 3692.704906] LNet: *** cfs_fail_loc=e003, val=100*** [ 3693.491282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.28@tcp [ 3693.512421] LNet: *** cfs_fail_loc=e003, val=100*** [ 3693.514340] LNet: Skipped 2 previous similar messages [ 3693.519844] Ping reply received with mapping: [ 3693.524156] found metadata reply NI[192.168.201.28@tcp] [ 3693.527566] 1c c9 a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 3693.530232] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 3693.533587] 5a 5a 5a 5a 5a 5a 5a 5a [ 3693.648855] LNet: 227665:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3693.654271] LNetError: 227665:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3693.668626] LNet: Removed LNI 192.168.201.28@tcp [ 3694.154979] Key type .llcrypt unregistered [ 3694.157477] Key type ._llcrypt unregistered [ 3707.149723] Key type ._llcrypt registered [ 3707.151798] Key type .llcrypt registered [ 3709.189759] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 11:57:24 (1788191844) [ 3709.797524] Key type .llcrypt unregistered [ 3709.800830] Key type ._llcrypt unregistered [ 3718.212879] Key type ._llcrypt registered [ 3718.215567] Key type .llcrypt registered [ 3718.696898] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 3718.744588] alg: No test for adler32 (adler32-zlib) [ 3719.935879] LNetError: 228715:0:(api-ni.c:2850:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 3720.629905] Key type .llcrypt unregistered [ 3720.635731] Key type ._llcrypt unregistered [ 3727.722858] Key type ._llcrypt registered [ 3727.728744] Key type .llcrypt registered [ 3729.409902] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 11:57:44 (1788191864) [ 3730.262501] Key type .llcrypt unregistered [ 3730.265409] Key type ._llcrypt unregistered [ 3737.893153] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 3737.901276] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 3740.563704] Key type ._llcrypt registered [ 3740.565895] Key type .llcrypt registered [ 3740.800572] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 3740.831870] alg: No test for adler32 (adler32-zlib) [ 3741.765172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3741.851988] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 3752.274302] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 11:58:07 (1788191887) [ 3752.593719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3752.614307] LNet: 230507:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3752.626669] LNetError: 230507:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 3752.640921] LNet: Removed LNI 10.1.2.3@tcp [ 3752.663635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3753.828830] Key type .llcrypt unregistered [ 3753.830768] Key type ._llcrypt unregistered [ 3761.211748] Key type ._llcrypt registered [ 3761.215645] Key type .llcrypt registered [ 3761.409880] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 3761.433307] alg: No test for adler32 (adler32-zlib) [ 3762.427495] LNet: Added LNI 192.168.201.28@tcp [8/256/0/180] [ 3762.529943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.28@tcp [ 3762.800252] LNet: 230793:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3762.807220] LNetError: 230793:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3762.823203] LNet: Removed LNI 192.168.201.28@tcp [ 3763.654590] Key type .llcrypt unregistered [ 3763.662325] Key type ._llcrypt unregistered [ 3777.558891] Key type ._llcrypt registered [ 3777.560828] Key type .llcrypt registered [ 3779.176566] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 11:58:34 (1788191914) [ 3779.919405] Key type .llcrypt unregistered [ 3779.920946] Key type ._llcrypt unregistered [ 3788.115878] Key type ._llcrypt registered [ 3788.118035] Key type .llcrypt registered [ 3788.483663] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 3788.522093] alg: No test for adler32 (adler32-zlib) [ 3798.140519] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 11:58:53 (1788191933) [ 3798.364650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3798.451434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3798.558873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 3798.599148] LNet: Added LNI 192.168.201.28@tcp [8/256/0/180] [ 3799.874828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 3799.926582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 3799.990221] LNet: Removed LNI 192.168.201.28@tcp [ 3800.457986] LNet: 232554:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3801.343466] Key type .llcrypt unregistered [ 3801.345120] Key type ._llcrypt unregistered [ 3815.698324] Key type ._llcrypt registered [ 3815.706444] Key type .llcrypt registered [ 3817.259226] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 11:59:13 (1788191953) [ 3818.131637] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3818.158930] alg: No test for adler32 (adler32-zlib) [ 3819.099175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3819.161490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 3819.196204] LNet: Added LNI 192.168.201.28@tcp [8/256/0/180] [ 3819.634474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.28@tcp [ 3819.652666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.201.28@tcp [ 3819.661899] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 3819.663955] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 3819.665402] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 3819, 0, 0 [ 3819.668083] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 3819.669531] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 3819.672453] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 3819.676416] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 3819, 0, 0 [ 3819.679991] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 3819.681899] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 3819.683690] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 3819.686363] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 3819, 0, 3869 [ 3819.688851] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 3819.691549] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 3819.694184] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 3819.695712] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 3819, 0, 3869 [ 3819.697990] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 3827.167323] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 11:59:22 (1788191962) [ 3827.853525] LNet: 234112:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3827.864757] LNetError: 234112:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3827.906063] LNet: Removed LNI 192.168.201.28@tcp [ 3828.930523] Key type .llcrypt unregistered [ 3828.948225] Key type ._llcrypt unregistered [ 3836.467590] Key type ._llcrypt registered [ 3836.470240] Key type .llcrypt registered [ 3836.965968] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 3837.023754] alg: No test for adler32 (adler32-zlib) [ 3837.972917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 3838.002586] LNet: Added LNI 192.168.201.28@tcp [8/256/0/180] [ 3847.259458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.128@tcp [ 3860.510835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3860.567097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 3860.586765] LNet: Added LNI 192.168.201.28@tcp1 [8/256/0/180] [ 3862.028309] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 3862.369859] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 3862.380892] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 3862.385806] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 3864.811443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 3864.840655] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 3878.445088] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 12:00:13 (1788192013) [ 3878.579919] LNet: 235619:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3878.589587] LNetError: 235619:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3878.597529] LNet: Removed LNI 192.168.201.28@tcp [ 3878.606183] LNet: Removed LNI 192.168.201.28@tcp1 [ 3878.626028] LNet: Removed LNI 10.1.2.3@tcp99 [ 3879.963891] Key type .llcrypt unregistered [ 3879.965325] Key type ._llcrypt unregistered [ 3887.450537] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 3887.458550] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 3890.406409] Key type ._llcrypt registered [ 3890.414383] Key type .llcrypt registered [ 3890.750682] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 3890.786689] alg: No test for adler32 (adler32-zlib) [ 3891.745404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3891.823560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 3891.849962] LNet: Added LNI 192.168.201.28@tcp [8/256/0/180] [ 3892.792495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 3892.802543] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 3895.790436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 3895.831865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 3895.861324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 3895.892599] LNetError: 236371:0:(peer.c:2362:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 3895.929620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 3895.972552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.28@tcp [ 3896.477288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 3896.622053] LNet: 236391:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3896.627923] LNetError: 236391:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 3896.633491] LNet: Removed LNI 10.1.2.3@tcp [ 3897.703876] LNet: Removed LNI 192.168.201.28@tcp [ 3898.399592] Key type .llcrypt unregistered [ 3898.401489] Key type ._llcrypt unregistered [ 3913.658124] Key type ._llcrypt registered [ 3913.660363] Key type .llcrypt registered [ 3915.150576] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 12:00:50 (1788192050) [ 3915.604680] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3915.624572] alg: No test for adler32 (adler32-zlib) [ 3916.628090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3916.701064] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 3916.722937] LNet: Added LNI 192.168.201.28@tcp [8/256/0/180] [ 3917.566576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.28@tcp [ 3917.637868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg128-client@tcp [ 3924.875327] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 12:01:00 (1788192060) [ 3925.531618] LNet: 237952:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3925.540165] LNetError: 237952:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3925.574589] LNet: Removed LNI 192.168.201.28@tcp [ 3926.410215] Key type .llcrypt unregistered [ 3926.417483] Key type ._llcrypt unregistered [ 3934.873950] Key type ._llcrypt registered [ 3934.884155] Key type .llcrypt registered [ 3935.105143] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3935.118174] alg: No test for adler32 (adler32-zlib) [ 3935.983825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 3936.024141] LNet: Added LNI 192.168.201.28@tcp [8/256/0/180] [ 3936.063590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 3944.170591] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 12:01:19 (1788192079) [ 3945.790611] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 3948.266427] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 12:01:23 (1788192083) [ 3950.314906] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 3952.446857] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 12:01:28 (1788192088) [ 3952.726218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3952.754309] LNet: 239391:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3952.762836] LNetError: 239391:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3952.780837] LNet: Removed LNI 192.168.201.28@tcp [ 3952.814594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3952.866596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 3952.900102] LNet: Added LNI 192.168.201.28@tcp [8/256/0/180] [ 3953.608469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 3953.676286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 3963.803514] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 12:01:38 (1788192098) [ 3963.938809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3963.974836] LNet: 240103:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3964.004251] LNetError: 240103:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3964.048581] LNet: Removed LNI 192.168.201.28@tcp [ 3964.099734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3964.176941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 3964.224758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 3965.014278] Key type .llcrypt unregistered [ 3965.020679] Key type ._llcrypt unregistered [ 3979.192739] Key type ._llcrypt registered [ 3979.195945] Key type .llcrypt registered [ 3981.205299] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 12:01:56 (1788192116) [ 3981.863230] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3981.885504] alg: No test for adler32 (adler32-zlib) [ 3982.982114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3983.081276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 3983.141780] LNet: Added LNI 192.168.201.28@tcp [8/256/0/180] [ 3984.136352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 3984.195888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.28@tcp [ 3992.852978] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 12:02:08 (1788192128) [ 3992.984917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3993.046373] LNet: 241674:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3993.062202] LNetError: 241674:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3993.090920] LNet: Removed LNI 192.168.201.28@tcp [ 3993.163175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3993.318827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 3993.381863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 4005.779813] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 12:02:20 (1788192140) [ 4005.945933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 4005.981790] LNet: 242265:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4006.017655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 4006.242695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 4007.078243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 4007.122442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 4007.278232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 4008.399940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 4008.534302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 4008.808991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 4008.874937] LNet: 242304:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4008.972943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 4009.134270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 4010.308582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 4010.339875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 4010.430753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 4010.886528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 4010.907805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 4020.954738] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 12:02:35 (1788192155) [ 4021.037540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 4021.071885] LNet: 242905:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4021.108766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 4021.190715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 4021.214026] LNet: Added LNI 192.168.201.28@tcp [8/256/0/180] [ 4022.106418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 4022.134490] LNet: Added LNI 192.168.201.28@tcp2 [8/256/0/180] [ 4023.071316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 4023.122485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.28@tcp [ 4023.304380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 4023.327680] LNetError: 243192:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4024.351192] LNet: Removed LNI 192.168.201.28@tcp [ 4024.362531] LNetError: 243192:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4024.398175] LNet: Removed LNI 192.168.201.28@tcp2 [ 4024.442162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 4024.573397] LNet: Added LNI 192.168.201.28@tcp [8/256/0/180] [ 4024.579968] LNet: Added LNI 192.168.201.28@tcp2 [8/256/0/180] [ 4024.597445] LNet: 243199:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 4025.459371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.28@tcp [ 4025.873883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 4025.913319] LNet: 243236:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4025.923109] LNet: 243236:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 4026.978714] LNet: Removed LNI 192.168.201.28@tcp [ 4026.984590] LNetError: 243236:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4026.994745] LNetError: 243236:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 4027.008177] LNet: Removed LNI 192.168.201.28@tcp2 [ 4027.084871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 4027.148475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 4027.180641] LNet: Added LNI 192.168.201.28@tcp [8/256/0/180] [ 4028.131258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 4028.167256] LNet: Added LNI 192.168.201.28@tcp2 [8/256/0/180] [ 4029.381696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.28@tcp [ 4029.432700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 4029.495285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.28@tcp [ 4029.812487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 4030.880185] LNet: Removed LNI 192.168.201.28@tcp [ 4031.913518] LNet: Removed LNI 192.168.201.28@tcp2 [ 4031.988319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 4032.108638] LNet: Added LNI 192.168.201.28@tcp [8/256/0/180] [ 4032.114632] LNet: Added LNI 192.168.201.28@tcp2 [8/256/0/180] [ 4032.123210] LNet: 243527:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 4032.814894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.28@tcp [ 4033.155436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 4033.191470] LNetError: 243564:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4033.197480] LNetError: 243564:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 4033.203680] LNet: Removed LNI 192.168.201.28@tcp [ 4034.220172] LNet: Removed LNI 192.168.201.28@tcp2 [ 4034.253151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 4034.352363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 4034.377930] LNet: Added LNI 192.168.201.28@tcp [8/256/0/180] [ 4035.135344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 4035.160579] LNet: Added LNI 192.168.201.28@tcp2 [8/256/0/180] [ 4035.930374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 4036.022375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.28@tcp [ 4036.196829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 4036.218890] LNet: 243843:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4036.224933] LNet: 243843:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 4037.279221] LNet: Removed LNI 192.168.201.28@tcp [ 4037.297255] LNet: Removed LNI 192.168.201.28@tcp2 [ 4037.358558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 4037.425039] LNet: Added LNI 192.168.201.28@tcp [8/256/0/180] [ 4037.430623] LNet: Added LNI 192.168.201.28@tcp2 [8/256/0/180] [ 4037.439946] LNet: 243849:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 4038.045725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.28@tcp [ 4038.289637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 4039.327172] LNet: Removed LNI 192.168.201.28@tcp [ 4039.339945] LNet: Removed LNI 192.168.201.28@tcp2 [ 4039.364794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 4039.455139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 4039.491523] LNet: Added LNI 192.168.201.28@tcp [8/256/0/180] [ 4040.354517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 4040.371674] LNet: Added LNI 192.168.201.28@tcp2 [8/256/0/180] [ 4041.103220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.28@tcp [ 4041.131885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 4041.179610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.28@tcp [ 4041.399244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 4041.422215] LNetError: 244171:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4041.425763] LNetError: 244171:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 4042.463174] LNet: Removed LNI 192.168.201.28@tcp [ 4043.499736] LNet: Removed LNI 192.168.201.28@tcp2 [ 4043.528818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 4043.614108] LNet: Added LNI 192.168.201.28@tcp [8/256/0/180] [ 4043.633629] LNet: Added LNI 192.168.201.28@tcp2 [8/256/0/180] [ 4043.640519] LNet: 244177:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 4044.540615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.28@tcp [ 4044.936973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 4044.976899] LNet: Removed LNI 192.168.201.28@tcp [ 4045.993951] LNet: Removed LNI 192.168.201.28@tcp2 [ 4046.022902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 4046.084462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 4046.121479] LNet: Added LNI 192.168.201.28@tcp [8/256/0/180] [ 4047.054878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 4047.077824] LNet: Added LNI 192.168.201.28@tcp2 [8/256/0/180] [ 4048.425900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 4048.465281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 4048.572787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.28@tcp [ 4049.155631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 4050.271197] LNet: Removed LNI 192.168.201.28@tcp [ 4050.288132] LNet: Removed LNI 192.168.201.28@tcp2 [ 4050.349205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 4050.443729] LNet: Added LNI 192.168.201.28@tcp [8/256/0/180] [ 4050.454226] LNet: Added LNI 192.168.201.28@tcp2 [8/256/0/180] [ 4050.459668] LNet: 244517:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 4051.343614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.28@tcp [ 4051.600756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 4052.639102] LNet: Removed LNI 192.168.201.28@tcp [ 4052.649678] LNet: Removed LNI 192.168.201.28@tcp2 [ 4052.666955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 4052.750426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 4052.861838] LNet: Added LNI 192.168.201.28@tcp [8/256/0/180] [ 4053.990930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 4054.046241] LNet: Added LNI 192.168.201.28@tcp2 [8/256/0/180] [ 4055.216587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.28@tcp [ 4055.252927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 4055.295907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 4055.381170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.28@tcp [ 4055.668768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 4055.684378] LNet: 244871:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4055.691508] LNet: 244871:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 5 previous similar messages [ 4056.736305] LNet: Removed LNI 192.168.201.28@tcp [ 4057.791227] LNet: Removed LNI 192.168.201.28@tcp2 [ 4057.852935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 4058.007788] LNet: Added LNI 192.168.201.28@tcp [8/256/0/180] [ 4058.021409] LNet: Added LNI 192.168.201.28@tcp2 [8/256/0/180] [ 4058.801056] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.28@tcp [ 4069.086693] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 4071.872980] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 12:03:27 (1788192207) [ 4072.007756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 4072.069445] LNetError: 245638:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4072.078788] LNetError: 245638:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 9 previous similar messages [ 4072.088827] LNet: Removed LNI 192.168.201.28@tcp [ 4073.141898] LNet: Removed LNI 192.168.201.28@tcp2 [ 4073.201283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 4073.891435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 4073.923526] LNet: Added LNI 192.168.201.28@tcp [8/256/0/180] [ 4074.666505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 4074.704170] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 4075.783224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 4075.805343] LNet: Removed LNI 10.1.2.3@tcp [ 4075.817077] LNet: Removed LNI 192.168.201.28@tcp [ 4075.841526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 4075.922949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 4075.958022] LNet: Added LNI 192.168.201.28@tcp [8/256/0/180] [ 4075.966785] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 4076.542627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 4076.576342] LNet: Removed LNI 192.168.201.28@tcp [ 4076.611368] LNet: Removed LNI 10.1.2.3@tcp [ 4084.484303] Lustre: DEBUG MARKER: == sanity-lnet test 475: per-operation latency stats populate and reset ========================================================== 12:03:40 (1788192220) [ 4084.776429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 4084.868819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 4084.965930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 4085.034519] LNet: Added LNI 192.168.201.28@tcp [8/256/0/180] [ 4086.129868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 4086.194513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.28@tcp [ 4086.231380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.28@tcp [ 4086.262275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.28@tcp [ 4086.329919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.28@tcp [ 4086.364424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.28@tcp [ 4086.394523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.28@tcp [ 4086.447357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.28@tcp [ 4086.510948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.28@tcp [ 4086.561340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.28@tcp [ 4086.621803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.28@tcp [ 4086.837412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 4093.742524] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 12:03:49 (1788192229) [ 4093.874394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 4093.893581] LNet: 247271:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4093.897258] LNet: 247271:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 4094.950921] LNet: Removed LNI 192.168.201.28@tcp [ 4095.001699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 4095.139322] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 4095.268843] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 4095.289613] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 4095.298406] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 4097.810840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 4097.857119] LNet: Added LNI 192.168.201.28@tcp [8/256/0/180] [ 4098.886891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 4098.905898] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 4100.941117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.28@tcp [ 4100.993740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 4101.076741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.201.28@tcp -m PUT -e local_timeout -r 1 [ 4101.156435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 4101.232986] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 4101.238187] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 4101.541905] LNetError: 247343:0:(lib-move.c:4801:lnet_parse()) 192.168.201.28@tcp, src 192.168.201.28@tcp: Dropping PUT (error -108 looking up sender) [ 4102.567209] LNet: Removed LNI 10.1.2.3@tcp [ 4103.597457] LNet: Removed LNI 192.168.201.28@tcp [ 4104.131878] Key type .llcrypt unregistered [ 4104.133958] Key type ._llcrypt unregistered [ 4119.440232] Key type ._llcrypt registered [ 4119.442466] Key type .llcrypt registered [ 4120.947860] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 12:04:16 (1788192256) [ 4121.302692] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4121.316939] alg: No test for adler32 (adler32-zlib) [ 4122.338852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 4122.775279] LNet: 248664:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4123.355485] Key type .llcrypt unregistered [ 4123.358434] Key type ._llcrypt unregistered [ 4130.766280] Key type ._llcrypt registered [ 4130.769986] Key type .llcrypt registered [ 4131.015905] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 4131.061202] alg: No test for adler32 (adler32-zlib) [ 4132.066830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 4132.093341] LNet: Added LNI 192.168.201.28@tcp [8/256/0/180] [ 4141.424624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.128@tcp [ 4141.742286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 4142.556771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.201.128@tcp [ 4142.791226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 4142.819304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 4142.848295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 4142.882525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 4142.911827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 4142.948333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 4142.987750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 4143.025499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 4143.043201] LNet: 249113:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 4143.060696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 4143.087528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 4143.116280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 4143.143740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 4143.169980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 4143.195365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 4143.802806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 4144.018543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 4144.048105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 4144.085553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 4144.108815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 4144.163233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 4144.194649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 4144.251543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 4144.317405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 4144.351882] LNet: 249230:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 4144.372952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 4144.420412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 4144.423213] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.128@tcp [ 4144.464928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 4144.510644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 4144.551539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 4144.594632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 4145.309091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.201.128@tcp [ 4145.614635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 4145.688743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 4145.757672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 4145.822760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 4145.875791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 4145.902378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 4145.947381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 4145.978229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 4145.989638] LNet: 249347:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 4146.011431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 4146.039115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 4146.082981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 4146.137267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 4146.176938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 4146.214968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 4146.832530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 4147.085656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 4147.117788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 4147.154438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 4147.203836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 4147.238912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 4147.293758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 4147.342780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 4147.403695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 4147.465234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 4147.525488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 4147.573766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 4147.612063] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 4147.656481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 4147.699377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 4148.041128] LNet: 249511:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4148.046987] LNetError: 249511:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4148.065478] LNet: Removed LNI 192.168.201.28@tcp [ 4148.477863] Key type .llcrypt unregistered [ 4148.480504] Key type ._llcrypt unregistered [ 4168.919405] Key type ._llcrypt registered [ 4168.922134] Key type .llcrypt registered [ 4170.742379] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 12:05:06 (1788192306) [ 4171.332132] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4171.356702] alg: No test for adler32 (adler32-zlib) [ 4172.488914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 4173.033894] LNet: 250408:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4173.953701] Key type .llcrypt unregistered [ 4173.958432] Key type ._llcrypt unregistered [ 4184.614148] Key type ._llcrypt registered [ 4184.618594] Key type .llcrypt registered [ 4185.024110] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 4185.073241] alg: No test for adler32 (adler32-zlib) [ 4186.072938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 4186.094969] LNet: Added LNI 192.168.201.28@tcp [8/256/0/180] [ 4195.228549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.128@tcp [ 4205.230895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 4205.295356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 4205.332180] LNet: Added LNI 192.168.201.28@tcp1 [8/256/0/180] [ 4206.718204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 4207.602925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.201.128@tcp [ 4207.648087] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.128@tcp [ 4207.924412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 4207.973729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 4208.012943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 4208.040196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 4208.081262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 4208.121280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 4208.198702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 4208.282608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 4208.327081] LNet: 251059:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 4208.360897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 4208.406520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 4208.454825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 4208.498484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 4208.565296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 4208.626905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 4209.644446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 4209.944930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 4209.990150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 4210.029556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 4210.068296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 4210.104939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 4210.142351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 4210.188316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 4210.225262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 4210.251756] LNet: 251177:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 4210.273723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 4210.310330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 4210.346375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 4210.390492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 4210.440146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 4210.476716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 4211.298308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.201.128@tcp [ 4211.580055] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 4211.610955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 4211.648273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 4211.696879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 4211.752962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 4211.821369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 4211.879644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 4211.935733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 4211.975462] LNet: 251294:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 4212.013954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 4212.061682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 4212.151600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 4212.232843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 4212.292287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 4212.361504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 4213.441740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 4213.764044] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 4213.814922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 4213.898313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 4213.954338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 4214.034339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 4214.087140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 4214.164324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 4214.204215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 4214.229340] LNet: 251411:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 4214.263353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 4214.333911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 4214.405513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 4214.456317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 4214.482863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 4214.512941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 4220.506271] LNet: 251492:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4220.511897] LNetError: 251492:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4220.520781] LNet: Removed LNI 192.168.201.28@tcp [ 4220.531453] LNet: Removed LNI 192.168.201.28@tcp1 [ 4221.099901] Key type .llcrypt unregistered [ 4221.102886] Key type ._llcrypt unregistered [ 4241.858113] Key type ._llcrypt registered [ 4241.862636] Key type .llcrypt registered [ 4243.769658] Lustre: DEBUG MARKER: == sanity-lnet test 525: keep reachable peer on local net delete ========================================================== 12:06:19 (1788192379) [ 4244.502936] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4244.539861] alg: No test for adler32 (adler32-zlib) [ 4245.681398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 4245.771996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 4245.808802] LNet: Added LNI 192.168.201.28@tcp [8/256/0/180] [ 4247.131697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 4247.163434] LNet: Added LNI 192.168.201.28@tcp2 [8/256/0/180] [ 4248.940658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.28@tcp [ 4249.060284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp [ 4250.146181] LNet: Removed LNI 192.168.201.28@tcp [ 4259.826937] Lustre: DEBUG MARKER: == sanity-lnet test 550: DD-off: keep route up when gw reachable via another local NI ========================================================== 12:06:35 (1788192395) [ 4261.729576] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_550 Need at least 2 local interfaces for multi-rail [ 4264.308456] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 12:06:39 (1788192399) [ 4272.565332] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 12:06:48 (1788192408) [ 4280.926916] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 12:06:56 (1788192416) [ 4290.672602] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 12:07:05 (1788192425) [ 4301.105037] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 12:07:17 (1788192437) [ 4314.115710] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 12:07:29 (1788192449) [ 4323.437721] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 12:07:39 (1788192459) [ 4333.765271] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 12:07:49 (1788192469) [ 4342.566530] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 12:07:58 (1788192478) [ 4354.169741] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 12:08:09 (1788192489) [ 4365.360933] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 12:08:20 (1788192500) [ 4375.569304] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 12:08:31 (1788192511) [ 4384.447030] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 12:08:40 (1788192520) [ 4393.697226] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 12:08:49 (1788192529) [ 4403.352289] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 12:08:59 (1788192539) [ 4414.529570] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 12:09:10 (1788192550) [ 4428.734670] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 12:09:22 (1788192562) [ 4441.752810] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 12:09:37 (1788192577) [ 4442.716156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 4442.801634] LNet: 263352:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4442.817380] LNetError: 263352:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4442.858669] LNet: Removed LNI 192.168.201.28@tcp2 [ 4442.941899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 4443.121218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 4443.170159] LNet: Added LNI 192.168.201.28@tcp [8/256/0/180] [ 4445.107920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 4445.133196] LNet: 263494:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4445.143337] LNetError: 263494:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4445.165336] LNet: Removed LNI 192.168.201.28@tcp [ 4445.192540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 4445.292624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 4445.331556] LNet: Added LNI 192.168.201.28@tcp [8/256/0/180] [ 4453.778288] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 12:09:49 (1788192589) [ 4462.776679] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 12:09:58 (1788192598) [ 4471.650900] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 12:10:07 (1788192607) [ 4480.880416] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 12:10:16 (1788192616) [ 4490.734461] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 12:10:26 (1788192626) [ 4501.020634] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 12:10:36 (1788192636) [ 4509.645461] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 12:10:45 (1788192645) [ 4517.462929] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 12:10:53 (1788192653) [ 4526.495798] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 12:11:02 (1788192662) [ 4534.622647] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 12:11:10 (1788192670) [ 4543.595818] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 12:11:19 (1788192679) [ 4552.041709] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 12:11:27 (1788192687) [ 4560.004841] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 4236 sec ========== 12:11:35 (1788192695) [ 4560.317251] LNet: 271068:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4560.329229] LNetError: 271068:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4560.354181] LNet: Removed LNI 192.168.201.28@tcp [ 4561.173187] Key type .llcrypt unregistered [ 4561.181783] Key type ._llcrypt unregistered [ 4577.042870] Key type ._llcrypt registered [ 4577.048913] Key type .llcrypt registered [ 4577.909124] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 4577.922362] alg: No test for adler32 (adler32-zlib) [ 4579.206945] Lustre: Lustre: Build Version: 2.17.57_85_gb71fc30 [ 4579.589747] LNet: Added LNI 192.168.201.28@tcp [8/256/0/180] [ 4581.295160] Key type lgssc registered [ 4583.585943] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4690.027714] Lustre: Mounted lustre-client - version 2.17.57_85_gb71fc30 [ 4696.359784] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4706.922260] Lustre: Unmounted lustre-client [ 4770.423257] Key type lgssc unregistered [ 4770.864877] LNet: 274584:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4770.873893] LNetError: 274584:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4770.906085] LNet: Removed LNI 192.168.201.28@tcp [ 4771.914872] Key type .llcrypt unregistered [ 4771.916671] Key type ._llcrypt unregistered