[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 467276291 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2672MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.002318] x2apic enabled [ 0.003009] Switched APIC routing to physical x2apic. [ 0.004015] kvm-guest: setup PV IPIs [ 0.007000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007038] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009016] pid_max: default: 32768 minimum: 301 [ 0.010167] LSM: Security Framework initializing [ 0.011052] Yama: becoming mindful. [ 0.012043] SELinux: Initializing. [ 0.014044] *** VALIDATE selinux *** [ 0.022410] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026535] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027134] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028113] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029097] *** VALIDATE tmpfs *** [ 0.030379] *** VALIDATE proc *** [ 0.031183] *** VALIDATE cgroup *** [ 0.032011] *** VALIDATE cgroup2 *** [ 0.033282] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.034151] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.035012] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.036028] Spectre V2 : User space: Vulnerable [ 0.037010] Speculative Store Bypass: Vulnerable [ 0.040194] debug: unmapping init [mem 0xffffffffbde59000-0xffffffffbde60fff] [ 0.042189] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.043712] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.044028] ... version: 2 [ 0.045017] ... bit width: 48 [ 0.046015] ... generic registers: 4 [ 0.047013] ... value mask: 0000ffffffffffff [ 0.048017] ... max period: 00007fffffffffff [ 0.049009] ... fixed-purpose events: 3 [ 0.049770] ... event mask: 000000070000000f [ 0.050330] rcu: Hierarchical SRCU implementation. [ 0.052536] smp: Bringing up secondary CPUs ... [ 0.053612] x86: Booting SMP configuration: [ 0.054036] .... node #0, CPUs: #1 #2 #3 [ 0.058015] smp: Brought up 1 node, 4 CPUs [ 0.060027] smpboot: Max logical packages: 1 [ 0.061020] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.148699] node 0 deferred pages initialised in 84ms [ 0.152158] devtmpfs: initialized [ 0.153283] x86/mm: Memory block size: 128MB [ 0.156785] gcov: version magic: 0x41383552 [ 0.159321] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.162156] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.165410] pinctrl core: initialized pinctrl subsystem [ 0.167191] [ 0.167701] ************************************************************* [ 0.169013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.172015] ** ** [ 0.174013] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.176016] ** ** [ 0.178013] ** This means that this kernel is built to expose internal ** [ 0.180014] ** IOMMU data structures, which may compromise security on ** [ 0.182012] ** your system. ** [ 0.185013] ** ** [ 0.187011] ** If you see this message and you are not debugging the ** [ 0.189013] ** kernel, report this immediately to your vendor! ** [ 0.191013] ** ** [ 0.193017] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.195012] ************************************************************* [ 0.197768] NET: Registered protocol family 16 [ 0.199430] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.201064] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.204086] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.206477] cpuidle: using governor menu [ 0.207545] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.210616] PCI: Using configuration type 1 for base access [ 0.212130] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.223069] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.225037] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.229035] cryptd: max_cpu_qlen set to 1000 [ 0.233848] ACPI: Added _OSI(Module Device) [ 0.235020] ACPI: Added _OSI(Processor Device) [ 0.239008] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.240023] ACPI: Added _OSI(Processor Aggregator Device) [ 0.245046] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.250576] ACPI: Interpreter enabled [ 0.252075] ACPI: PM: (supports S0 S3 S4 S5) [ 0.253014] ACPI: Using IOAPIC for interrupt routing [ 0.254111] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.257410] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.267177] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.269049] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.271027] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.274109] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.279000] acpiphp: Slot [2] registered [ 0.279000] acpiphp: Slot [5] registered [ 0.281120] acpiphp: Slot [6] registered [ 0.281877] acpiphp: Slot [3] registered [ 0.283119] acpiphp: Slot [4] registered [ 0.285103] acpiphp: Slot [7] registered [ 0.286110] acpiphp: Slot [8] registered [ 0.287118] acpiphp: Slot [9] registered [ 0.288095] acpiphp: Slot [10] registered [ 0.290141] acpiphp: Slot [11] registered [ 0.291097] acpiphp: Slot [12] registered [ 0.292134] acpiphp: Slot [13] registered [ 0.293114] acpiphp: Slot [14] registered [ 0.295111] acpiphp: Slot [15] registered [ 0.296133] acpiphp: Slot [16] registered [ 0.297110] acpiphp: Slot [17] registered [ 0.299120] acpiphp: Slot [18] registered [ 0.300155] acpiphp: Slot [19] registered [ 0.301103] acpiphp: Slot [20] registered [ 0.303100] acpiphp: Slot [21] registered [ 0.304092] acpiphp: Slot [22] registered [ 0.305119] acpiphp: Slot [23] registered [ 0.307131] acpiphp: Slot [24] registered [ 0.308101] acpiphp: Slot [25] registered [ 0.310144] acpiphp: Slot [26] registered [ 0.311100] acpiphp: Slot [27] registered [ 0.312105] acpiphp: Slot [28] registered [ 0.314125] acpiphp: Slot [29] registered [ 0.315145] acpiphp: Slot [30] registered [ 0.317104] acpiphp: Slot [31] registered [ 0.318086] PCI host bridge to bus 0000:00 [ 0.320017] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.322023] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.324027] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.325018] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.327017] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.329026] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.331264] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.334156] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.337323] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.345016] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.349122] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.352025] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.354020] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.356025] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.358717] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.361844] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.364050] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.366950] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.371016] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.380738] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.385023] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.391446] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.405021] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.413030] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.432026] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.442452] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.449020] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.458023] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.480022] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.495792] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.498610] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.501557] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.504459] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.506319] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.513044] iommu: Default domain type: Passthrough [ 0.514456] SCSI subsystem initialized [ 0.515209] ACPI: bus type USB registered [ 0.516132] usbcore: registered new interface driver usbfs [ 0.518114] usbcore: registered new interface driver hub [ 0.521112] usbcore: registered new device driver usb [ 0.522213] pps_core: LinuxPPS API ver. 1 registered [ 0.524013] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.528082] PTP clock support registered [ 0.530060] EDAC MC: Ver: 3.0.0 [ 0.532268] PCI: Using ACPI for IRQ routing [ 0.534916] NetLabel: Initializing [ 0.536020] NetLabel: domain hash size = 128 [ 0.537016] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.539128] NetLabel: unlabeled traffic allowed by default [ 0.541217] vgaarb: loaded [ 0.543338] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.545030] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.550465] clocksource: Switched to clocksource kvm-clock [ 0.662759] VFS: Disk quotas dquot_6.6.0 [ 0.664484] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.667158] *** VALIDATE ramfs *** [ 0.668175] *** VALIDATE hugetlbfs *** [ 0.669927] pnp: PnP ACPI init [ 0.672489] pnp: PnP ACPI: found 6 devices [ 0.691452] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.695305] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.698190] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.700434] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.702814] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.705419] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.708682] NET: Registered protocol family 2 [ 0.711456] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.716965] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.721057] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.726777] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.730564] TCP: Hash tables configured (established 65536 bind 65536) [ 0.735893] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.739401] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.742464] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.745811] NET: Registered protocol family 1 [ 0.750423] RPC: Registered named UNIX socket transport module. [ 0.752847] RPC: Registered udp transport module. [ 0.754743] RPC: Registered tcp transport module. [ 0.756515] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.759086] NET: Registered protocol family 44 [ 0.761067] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.763589] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.765975] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.768309] PCI: CLS 0 bytes, default 64 [ 0.770258] Unpacking initramfs... [ 2.229994] debug: unmapping init [mem 0xffff98cffcc64000-0xffff98cffffcffff] [ 2.234284] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.236833] software IO TLB: mapped [mem 0x00000000a3000000-0x00000000a7000000] (64MB) [ 2.239959] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.772969] Initialise system trusted keyrings [ 2.774757] Key type blacklist registered [ 2.777193] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.787275] zbud: loaded [ 2.790508] *** VALIDATE nfs *** [ 2.791649] *** VALIDATE nfs4 *** [ 2.792994] pstore: using deflate compression [ 2.796707] Platform Keyring initialized [ 2.914609] NET: Registered protocol family 38 [ 2.916944] Key type asymmetric registered [ 2.918488] Asymmetric key parser 'x509' registered [ 2.920185] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.922717] io scheduler mq-deadline registered [ 2.924279] io scheduler kyber registered [ 2.925938] io scheduler bfq registered [ 2.928246] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.931746] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.934964] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.937286] ACPI: Power Button [PWRF] [ 2.942427] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.949523] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.981256] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.009044] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.037960] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.043207] Non-volatile memory driver v1.3 [ 3.044846] Linux agpgart interface v0.103 [ 3.080587] virtio_blk virtio1: [vda] 149944 512-byte logical blocks (76.8 MB/73.2 MiB) [ 3.083717] vda: detected capacity change from 0 to 76771328 [ 3.098891] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.101142] vdb: detected capacity change from 0 to 1073741824 [ 3.107054] libphy: Fixed MDIO Bus: probed [ 3.112391] usbcore: registered new interface driver usbserial_generic [ 3.114029] usbserial: USB Serial support registered for generic [ 3.115672] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.119557] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.121165] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.123556] mousedev: PS/2 mouse device common for all mice [ 3.126596] rtc_cmos 00:05: RTC can wake from S4 [ 3.128857] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.130473] rtc_cmos 00:05: registered as rtc0 [ 3.134755] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.135331] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.137901] intel_pstate: CPU model not supported [ 3.142909] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.150872] hid: raw HID events driver (C) Jiri Kosina [ 3.153360] usbcore: registered new interface driver usbhid [ 3.155513] usbhid: USB HID core driver [ 3.157368] drop_monitor: Initializing network drop monitor service [ 3.159976] Initializing XFRM netlink socket [ 3.162061] NET: Registered protocol family 10 [ 3.164484] Segment Routing with IPv6 [ 3.166034] NET: Registered protocol family 17 [ 3.168539] mpls_gso: MPLS GSO support [ 3.174333] RAS: Correctable Errors collector initialized. [ 3.175844] AVX version of gcm_enc/dec engaged. [ 3.177133] AES CTR mode by8 optimization enabled [ 3.262108] sched_clock: Marking stable (3262037834, 0)->(4150588683, -888550849) [ 3.265509] registered taskstats version 1 [ 3.267400] Loading compiled-in X.509 certificates [ 3.269454] zswap: loaded using pool lzo/zbud [ 3.299498] Key type big_key registered [ 3.313529] Key type encrypted registered [ 3.315681] ima: No TPM chip found, activating TPM-bypass! [ 3.317587] ima: Allocated hash algorithm: sha1 [ 3.319355] ima: No architecture policies found [ 3.321335] evm: Initialising EVM extended attributes: [ 3.322818] evm: security.selinux [ 3.323986] evm: security.ima [ 3.325252] evm: security.capability [ 3.326382] evm: HMAC attrs: 0x1 [ 3.328646] rtc_cmos 00:05: setting system clock to 2026-09-16 22:07:09 UTC (1789596429) [ 3.335346] debug: unmapping init [mem 0xffffffffbee03000-0xffffffffbeffffff] [ 3.338705] debug: unmapping init [mem 0xffffffffbdb82000-0xffffffffbde58fff] [ 3.350247] Write protecting the kernel read-only data: 28672k [ 3.353827] debug: unmapping init [mem 0xffffffffbc203000-0xffffffffbc3fffff] [ 3.357033] debug: unmapping init [mem 0xffffffffbcb14000-0xffffffffbcbfffff] [ 3.426182] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.432983] systemd[1]: Detected virtualization kvm. [ 3.434617] systemd[1]: Detected architecture x86-64. [ 3.436182] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.463768] systemd[1]: No hostname configured. [ 3.465430] systemd[1]: Set hostname to . [ 3.467259] random: systemd: uninitialized urandom read (16 bytes read) [ 3.469599] systemd[1]: Initializing machine ID from random generator. [ 3.529714] random: ln: uninitialized urandom read (6 bytes read) [ 3.625835] random: systemd: uninitialized urandom read (16 bytes read) [ 3.628119] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 3.631500] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 3.635907] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. Starting Setup Virtual Console... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Local File Systems. [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on udev Control Socket. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. Starting Journal Service... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Paths. Starting Create Volatile Files and Directories... [ OK ] Reached target Swap. Starting Apply Kernel Variables... [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.294407] device-mapper: uevent: version 1.0.3 [ 4.296187] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.009724] virtio_net virtio0 ens2: renamed from eth0 [ 5.059931] scsi host0: ata_piix [ 5.080667] scsi host1: ata_piix [ 5.082021] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 5.084291] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 6.625227] random: fast init done [ 8.817807] dracut-initqueue[586]: RTNETLINK answers: File exists [ 9.831190] random: crng init done [ 9.831974] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.262736] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.434901] printk: systemd: 25 output lines suppressed due to ratelimiting [ 11.638991] SELinux: Disabled at runtime. [ 11.701596] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.709519] systemd[1]: Detected virtualization kvm. [ 11.711192] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.177540] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.180827] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.185773] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.190246] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.193880] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.203429] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.213349] systemd[1]: Mounting Kernel Debug File System... Mounting Kernel Debug File System... [ OK ] Created slice system-serial\x2dgetty.slice. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target rpc_pipefs.target. Starting Apply Kernel Variables... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Reached target RPC Port Mapper. [ OK ] Started Forward Password Requests to Wall Directory Watch. Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on Process Core Dump Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ 12.293990] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Mounting POSIX Message Queue File System... [ OK ] Reached target Local Encrypted Volumes. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. Mounting Huge Pages File System... [ OK ] Created slice system-getty.slice. Starting Remount Root and Kernel File Systems... [ OK ] Stopped target Initrd Root File System. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Started Journal Service. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Huge Pages File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 12.614744] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Coldplug all Devices. [ OK ] Started udev Kernel Device Manager. [ 12.983542] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 12.999058] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.181320] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.191704] EDAC sbridge: Ver: 1.1.2 [ 14.436526] Key type dns_resolver registered [ 14.755043] NFS: Registering the id_resolver key type [ 14.757225] Key type id_resolver registered [ 14.758867] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Load/Save Random Seed. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. Starting Login Service... Starting Network Manager... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started irqbalance daemon. [ OK ] Started daily update of the root trust anchor for DNSSEC. Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg116-client login: [ 29.079080] hrtimer: interrupt took 2069324 ns [ 87.928759] libcfs: loading out-of-tree module taints kernel. [ 88.255973] Key type ._llcrypt registered [ 88.263206] Key type .llcrypt registered [ 88.916123] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 88.952981] alg: No test for adler32 (adler32-zlib) [ 90.471210] Lustre: Lustre: Build Version: 2.17.58_39_g6aa67ea [ 91.768082] LNet: Added LNI 192.168.201.16@tcp [8/256/0/180] [ 93.600305] Key type lgssc registered [ 96.213440] Lustre: Echo OBD driver; http://www.lustre.org/ [ 249.364915] Lustre: Mounted lustre-client - version 2.17.58_39_g6aa67ea [ 254.710207] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 267.199870] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing check_logdir /tmp/testlogs/ [ 274.427149] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing yml_node [ 274.918364] Lustre: lustre-OST0000-osc-ffff98d048f98800: disconnect after 24s idle [ 279.307515] Lustre: DEBUG MARKER: Client: 2.17.58.39 [ 282.170875] Lustre: DEBUG MARKER: MDS: 2.17.58.39 [ 285.180577] Lustre: DEBUG MARKER: OSS: 2.17.58.39 [ 287.051472] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Wed Sep 16 18:11:51 EDT 2026 [ 305.400902] Lustre: Unmounted lustre-client [ 330.766265] Key type lgssc unregistered [ 331.012165] LNet: 8409:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 331.022299] LNetError: 8409:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 332.085609] LNet: Removed LNI 192.168.201.16@tcp [ 333.079168] Key type .llcrypt unregistered [ 333.085568] Key type ._llcrypt unregistered [ 363.733093] Key type ._llcrypt registered [ 363.740048] Key type .llcrypt registered [ 363.961432] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 363.975269] alg: No test for adler32 (adler32-zlib) [ 364.934860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 364.997891] LNet: Added LNI 192.168.201.16@tcp [8/256/0/180] [ 365.016512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 370.327966] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 370.686771] LNet: 9292:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 370.690843] LNetError: 9292:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 370.703231] LNet: Removed LNI 192.168.201.16@tcp [ 371.634475] Key type .llcrypt unregistered [ 371.636964] Key type ._llcrypt unregistered [ 379.728955] Key type ._llcrypt registered [ 379.732752] Key type .llcrypt registered [ 382.077806] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 18:13:25 (1789596805) [ 382.394703] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 382.404914] alg: No test for adler32 (adler32-zlib) [ 383.416717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 383.549679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 383.588622] LNet: 9700:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 383.614644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 383.664828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 383.683358] LNet: 9707:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 392.509574] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 18:13:36 (1789596816) [ 392.651855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 392.678639] LNet: 10281:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 392.691967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 392.800384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 401.545475] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 18:13:45 (1789596825) [ 402.072988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 402.111615] LNet: 10872:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 402.215936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 402.487648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 412.558845] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 18:13:56 (1789596836) [ 413.121352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 413.156654] LNet: 11457:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 413.190323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 413.292280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 421.414294] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 18:14:05 (1789596845) [ 421.539036] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 421.586144] LNet: 12033:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 421.628363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 421.824183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 422.359395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 432.025624] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 18:14:16 (1789596856) [ 432.116198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 432.149319] LNet: 12627:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 432.194271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 432.280691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 432.536550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 432.788657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 440.607090] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 18:14:25 (1789596865) [ 440.798357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 440.920138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 441.000387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 448.564352] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 18:14:33 (1789596873) [ 448.707316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 448.735745] LNet: 13811:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 448.738707] LNet: 13811:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 448.784587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 448.831866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 448.858916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 449.346387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 449.421706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 449.972234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 450.041751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 450.590790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 450.632475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 451.010225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 451.095993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 451.510551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 451.553390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 452.202421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 452.253310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 452.695794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 452.727757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 453.150743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 453.187443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 453.492537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 453.512681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 453.770451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 453.796523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 454.252920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 454.291739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 454.601646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 454.633547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 462.409357] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 18:14:46 (1789596886) [ 462.670674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 462.725181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 462.800374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 462.829759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 469.849950] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 18:14:54 (1789596894) [ 470.015047] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 470.079709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 470.144139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 470.167809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 477.523410] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 18:15:01 (1789596901) [ 477.711993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 477.750972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 477.857286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 477.887157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 487.023492] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 18:15:10 (1789596910) [ 487.129701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 487.194660] LNet: 16247:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 487.198251] LNet: 16247:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 487.267915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 487.425964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 487.484737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 496.420207] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 18:15:20 (1789596920) [ 496.520680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 496.558940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 496.698179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 496.770570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 504.519435] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 18:15:29 (1789596929) [ 504.813085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 504.859031] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 504.914948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 504.950801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 511.382570] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 18:15:36 (1789596936) [ 511.771749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 511.840254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 511.929957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 512.521816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 519.613461] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 18:15:44 (1789596944) [ 519.866374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 519.918545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 520.090805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 520.350160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 527.211817] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 18:15:52 (1789596952) [ 527.545297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 527.596737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 527.800243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 528.043409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 534.766908] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 18:15:59 (1789596959) [ 535.062154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 535.104465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 535.513203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 535.822769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 544.491847] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 18:16:08 (1789596968) [ 545.372651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 545.435766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 545.570281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 545.646797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 554.939308] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 18:16:19 (1789596979) [ 555.251786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 555.302376] LNet: 21089:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 555.310647] LNet: 21089:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 7 previous similar messages [ 555.328085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 555.376536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 555.423451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 565.913544] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 18:16:29 (1789596989) [ 566.042522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 566.119794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 566.239483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 574.555193] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 18:16:39 (1789596999) [ 574.860919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 574.898484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 575.042416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 582.989917] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 18:16:47 (1789597007) [ 583.330321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 583.412673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 583.530729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 583.916805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 591.107483] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 18:16:55 (1789597015) [ 591.318403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 591.379762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 591.523757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 591.590817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 599.507791] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 18:17:03 (1789597023) [ 599.675593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 599.781853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 599.984519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 600.023539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 608.597109] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 18:17:13 (1789597033) [ 608.681885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 608.743729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 608.850964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 608.911474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 616.863771] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 18:17:21 (1789597041) [ 617.023588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 617.159138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 617.301626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 617.377920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 623.950635] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 18:17:28 (1789597048) [ 624.216859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 624.274610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 624.327377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 624.406919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 624.435695] LNetError: 25817:0:(peer.c:2366:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 624.461929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 624.500082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 631.393510] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 18:17:36 (1789597056) [ 631.774201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 631.855431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 632.042308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.KoJjsguLnP [ 632.149544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.KoJjsguLnP [ 632.263125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.KoJjsguLnP [ 640.953807] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 18:17:45 (1789597065) [ 641.234984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 641.293749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 641.374366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 641.421335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 641.464848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 648.709084] Lustre: DEBUG MARKER: == sanity-lnet test 40: LNetAddPeer merges a pre-existing non-primary peer NID ========================================================== 18:17:53 (1789597073) [ 648.798264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 648.844687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 648.966804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 648.989421] LNet: Added LNI 192.168.201.16@tcp [8/256/0/180] [ 650.611765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.16@tcp [ 652.351850] Lustre: Lustre: Build Version: 2.17.58_39_g6aa67ea [ 654.440513] Key type lgssc registered [ 655.766612] Lustre: Echo OBD driver; http://www.lustre.org/ [ 660.422469] LNetError: 27577:0:(lib-move.c:2151:lnet_handle_find_routed_path()) peer 192.168.201.16@tcp99 has no available nets [ 663.301437] Key type lgssc unregistered [ 663.587223] LNetError: 28683:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 663.600725] LNet: Removed LNI 192.168.201.16@tcp [ 664.225584] Key type .llcrypt unregistered [ 664.231730] Key type ._llcrypt unregistered [ 671.440810] Key type ._llcrypt registered [ 671.443852] Key type .llcrypt registered [ 672.740161] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 18:18:17 (1789597097) [ 673.224311] Key type .llcrypt unregistered [ 673.228867] Key type ._llcrypt unregistered [ 680.093428] Key type ._llcrypt registered [ 680.097320] Key type .llcrypt registered [ 680.399526] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 680.437942] alg: No test for adler32 (adler32-zlib) [ 681.451244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 681.604830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 681.643749] LNet: 29604:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=16 less than recommended minimum 512 [ 681.651184] LNet: 29604:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 [ 681.657996] LNet: 29604:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 [ 681.673756] LNet: Message forwarding will be enabled in 2s [ 684.768215] LNet: Message forwarding enabled [ 685.385615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 685.433611] LNet: Message forwarding will stop in 2s to 6s [ 687.851753] LNet: Message forwarding disabled [ 689.157711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 689.184716] LNet: 29693:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=16 less than recommended minimum 512 [ 689.198250] LNet: 29693:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 [ 689.210562] LNet: 29693:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 [ 689.225856] LNet: Message forwarding will be enabled in 2s [ 689.451195] LNet: 29706:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 690.376582] Key type .llcrypt unregistered [ 690.379491] Key type ._llcrypt unregistered [ 704.569465] Key type ._llcrypt registered [ 704.574155] Key type .llcrypt registered [ 705.903347] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 18:18:50 (1789597130) [ 706.455254] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 706.479071] alg: No test for adler32 (adler32-zlib) [ 707.413370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 707.487141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 714.577340] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 18:18:59 (1789597139) [ 714.999542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 715.017736] LNet: 31108:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 715.035110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 715.074164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 715.100434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 715.157505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 715.211647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 715.254524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 715.283906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 715.306314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 715.331971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 715.368847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 715.402485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 715.459813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 715.499786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 715.541656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 715.585616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 715.622677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 715.686303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 715.736501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 715.765304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 715.784355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 715.793906] LNetError: 31151:0:(router.c:625:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 723.116037] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 18:19:07 (1789597147) [ 723.236535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 723.276362] LNet: 31715:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 723.308947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 723.415986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 731.521146] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 18:19:16 (1789597156) [ 734.240925] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg116-server [ 737.051805] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 18:19:21 (1789597161) [ 737.483310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 737.524258] LNet: 32656:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 737.578188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 737.707108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 737.768787] LNet: Added LNI 192.168.201.16@tcp [8/256/0/180] [ 738.655741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.201.[17-21]@tcp [ 738.697457] LNet: 32789:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 738.777035] LNetError: 32660:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.201.17@tcp is being used as a gateway but routing feature is not turned on [ 738.814515] LNetError: 32660:0:(lib-lnet.h:1281:lnet_set_route_aliveness()) route to tcp1 through 192.168.201.17@tcp has gone from up to down [ 746.867292] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 18:19:31 (1789597171) [ 748.734244] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg116-server [ 750.692492] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 18:19:35 (1789597175) [ 750.892293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 750.927180] LNet: 33723:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 750.941277] LNetError: 33723:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 750.992263] LNet: Removed LNI 192.168.201.16@tcp [ 751.060883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 751.105676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 751.125458] LNet: Added LNI 192.168.201.16@tcp [8/256/0/180] [ 751.710939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.201.[17-22/2]@tcp [ 751.749188] LNet: 33843:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 751.763183] LNet: 33843:0:(router.c:729:lnet_add_route()) Skipped 4 previous similar messages [ 751.798343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.201.[17-22/2]@tcp [ 751.818990] LNetError: 33730:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.201.17@tcp is being used as a gateway but routing feature is not turned on [ 751.851502] LNetError: 33730:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 751.857645] LNetError: 33730:0:(lib-lnet.h:1281:lnet_set_route_aliveness()) route to tcp103 through 192.168.201.17@tcp has gone from up to down [ 751.865533] LNetError: 33730:0:(lib-lnet.h:1281:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 759.490688] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 18:19:44 (1789597184) [ 759.919436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 759.958170] LNet: 34436:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 759.964779] LNetError: 34436:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 759.982594] LNet: Removed LNI 192.168.201.16@tcp [ 760.006447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 760.056510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 760.094648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 760.120984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 760.176888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 760.227707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 760.258983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 760.304706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 760.392876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 760.429152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 760.527284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 760.632530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 760.692514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 760.771962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 760.828554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 760.898239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 761.061786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 761.195228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 761.240739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 761.331603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 761.424482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 761.494465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 761.594032] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 761.684726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 761.710295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 761.751650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 761.811225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 761.845279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 761.903333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 762.021970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 762.058150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 762.096960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 762.148535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 762.179497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 762.203682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 768.765432] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 18:19:53 (1789597193) [ 770.888161] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg116-server [ 772.580269] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 18:19:57 (1789597197) [ 774.489761] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg116-server [ 776.202619] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 18:20:01 (1789597201) [ 776.574848] LNet: 35836:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 776.581402] LNet: 35836:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 11 previous similar messages [ 777.232690] Key type .llcrypt unregistered [ 777.235037] Key type ._llcrypt unregistered [ 783.419504] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 783.627070] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 783.631115] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 783.641499] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 786.276658] Key type ._llcrypt registered [ 786.279891] Key type .llcrypt registered [ 786.523861] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 786.547255] alg: No test for adler32 (adler32-zlib) [ 787.485085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 787.555483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 787.596242] LNet: Added LNI 192.168.201.16@tcp [8/256/0/180] [ 788.330454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 788.346766] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 790.025347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 790.043599] LNet: Removed LNI 10.1.2.3@tcp [ 790.143686] LNet: 36550:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 790.149553] LNetError: 36550:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 790.165792] LNet: Removed LNI 192.168.201.16@tcp [ 790.798781] Key type .llcrypt unregistered [ 790.801136] Key type ._llcrypt unregistered [ 803.501817] Key type ._llcrypt registered [ 803.504525] Key type .llcrypt registered [ 805.150566] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 18:20:29 (1789597229) [ 805.824350] Key type .llcrypt unregistered [ 805.826915] Key type ._llcrypt unregistered [ 811.667567] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 811.957871] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 811.960232] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 811.982859] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 815.080845] Key type ._llcrypt registered [ 815.082479] Key type .llcrypt registered [ 815.369991] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 815.406259] alg: No test for adler32 (adler32-zlib) [ 816.451623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 816.570583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 816.608857] LNet: Added LNI 192.168.201.16@tcp [8/256/0/180] [ 818.232984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 818.278889] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 821.450949] LNet: 38041:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 821.458877] LNetError: 38041:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 821.465689] LNet: Removed LNI 10.1.2.3@tcp [ 821.474849] LNet: Removed LNI 192.168.201.16@tcp [ 822.838775] Key type .llcrypt unregistered [ 822.843972] Key type ._llcrypt unregistered [ 841.165518] Key type ._llcrypt registered [ 841.167226] Key type .llcrypt registered [ 842.954280] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 18:21:07 (1789597267) [ 843.514749] Key type .llcrypt unregistered [ 843.517971] Key type ._llcrypt unregistered [ 850.545818] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 850.682150] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 850.697317] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 850.705160] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 853.350170] Key type ._llcrypt registered [ 853.352831] Key type .llcrypt registered [ 853.564185] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 853.577571] alg: No test for adler32 (adler32-zlib) [ 854.550366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 854.615063] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 854.687070] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 857.204445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 857.235503] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 859.307336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 859.326415] LNet: Removed LNI 10.1.2.3@tcp [ 859.337622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 859.357969] LNet: Removed LNI 10.1.2.31@tcp [ 859.371494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 859.402082] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 860.272728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 860.293597] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 861.262979] LNet: 39951:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 861.278530] LNetError: 39951:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg:0 not found [ 861.287603] LNet: Removed LNI 10.1.2.31@tcp [ 861.311202] LNet: Removed LNI 10.1.2.3@tcp [ 862.271309] Key type .llcrypt unregistered [ 862.278314] Key type ._llcrypt unregistered [ 876.118398] Key type ._llcrypt registered [ 876.121672] Key type .llcrypt registered [ 877.648867] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 18:21:42 (1789597302) [ 878.462956] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 878.475734] alg: No test for adler32 (adler32-zlib) [ 879.500081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 879.570765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 879.599102] LNet: Added LNI 192.168.201.16@tcp [8/256/0/180] [ 880.584754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 880.601993] LNet: 40901:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 880.612574] LNetError: 40901:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 880.627120] LNet: Removed LNI 192.168.201.16@tcp [ 880.668046] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 880.743763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.201.16@tcp [ 880.805415] LNet: Added LNI 192.168.201.16@tcp [8/256/0/180] [ 881.831511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.201.16@tcp [ 881.881407] LNet: Removed LNI 192.168.201.16@tcp [ 881.992909] LNet: 41036:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 882.772600] Key type .llcrypt unregistered [ 882.785472] Key type ._llcrypt unregistered [ 899.348407] Key type ._llcrypt registered [ 899.357380] Key type .llcrypt registered [ 901.538994] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 18:22:05 (1789597325) [ 904.200610] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg116-server [ 908.222449] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 18:22:11 (1789597331) [ 909.108400] Key type .llcrypt unregistered [ 909.111238] Key type ._llcrypt unregistered [ 918.255400] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 918.767062] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 918.774405] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 918.793205] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 921.425780] Key type ._llcrypt registered [ 921.427429] Key type .llcrypt registered [ 921.623796] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 921.638479] alg: No test for adler32 (adler32-zlib) [ 922.751572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 922.840521] LNet: Added LNI 192.168.201.16@tcp1 [8/256/0/180] [ 924.062628] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 927.214670] LNet: 42865:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 927.218655] LNetError: 42865:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 927.221553] LNet: Removed LNI 192.168.201.16@tcp1 [ 927.229975] LNet: Removed LNI 10.1.2.3@tcp2 [ 927.964967] Key type .llcrypt unregistered [ 927.969492] Key type ._llcrypt unregistered [ 943.959972] Key type ._llcrypt registered [ 943.963234] Key type .llcrypt registered [ 945.880414] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 18:22:49 (1789597369) [ 946.436321] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 946.462035] alg: No test for adler32 (adler32-zlib) [ 947.599427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 947.698139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 947.731928] LNet: Added LNI 192.168.201.16@tcp [8/256/0/180] [ 948.547761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 948.588667] LNet: 43833:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 948.596444] LNetError: 43833:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 948.619408] LNet: Removed LNI 192.168.201.16@tcp [ 948.634502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 948.701030] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 948.735980] LNet: Added LNI 192.168.201.16@tcp [8/256/0/180] [ 955.576977] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 18:23:00 (1789597380) [ 955.890599] LNet: 44424:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 955.923161] LNetError: 44424:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 955.944363] LNet: Removed LNI 192.168.201.16@tcp [ 956.487618] Key type .llcrypt unregistered [ 956.496674] Key type ._llcrypt unregistered [ 964.013111] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 964.461784] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 964.475322] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 964.483149] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 967.174381] Key type ._llcrypt registered [ 967.178490] Key type .llcrypt registered [ 967.533246] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 967.566981] alg: No test for adler32 (adler32-zlib) [ 968.512366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 968.570165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 968.593347] LNet: Added LNI 192.168.201.16@tcp [8/256/0/180] [ 969.441796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 969.471987] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 971.756751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 971.791716] LNet: 45163:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 971.803114] LNetError: 45163:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 971.811301] LNet: Removed LNI 10.1.2.3@tcp [ 971.829254] LNet: Removed LNI 192.168.201.16@tcp [ 971.882666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 971.995793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 972.030254] LNet: Added LNI 192.168.201.16@tcp [8/256/0/180] [ 972.037598] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 972.659919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 972.684209] LNet: 45191:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 972.690828] LNetError: 45191:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 972.694310] LNetError: 45191:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 972.698194] LNet: Removed LNI 10.1.2.3@tcp [ 972.707207] LNet: Removed LNI 192.168.201.16@tcp [ 972.724707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 972.806703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 972.828241] LNet: Added LNI 192.168.201.16@tcp [8/256/0/180] [ 972.830686] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 973.266958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 973.293995] LNet: Added LNI 192.168.201.16@tcp2 [8/256/0/180] [ 974.351114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 974.374619] LNet: 45353:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 974.381299] LNetError: 45353:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 974.386277] LNetError: 45353:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 974.394684] LNet: Removed LNI 10.1.2.3@tcp [ 974.396493] LNet: Removed LNI 192.168.201.16@tcp [ 974.405188] LNet: Removed LNI 192.168.201.16@tcp2 [ 974.425536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 974.480529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 974.515544] LNet: Added LNI 192.168.201.16@tcp [8/256/0/180] [ 974.519610] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 974.525468] LNet: Added LNI 192.168.201.16@tcp2 [8/256/0/180] [ 982.732831] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 18:23:27 (1789597407) [ 983.092745] LNet: 45951:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 983.107074] LNetError: 45951:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 983.117222] LNetError: 45951:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 983.128050] LNet: Removed LNI 10.1.2.3@tcp [ 983.129778] LNet: Removed LNI 192.168.201.16@tcp [ 983.146450] LNet: Removed LNI 192.168.201.16@tcp2 [ 983.791175] Key type .llcrypt unregistered [ 983.793155] Key type ._llcrypt unregistered [ 990.781165] Key type ._llcrypt registered [ 990.783308] Key type .llcrypt registered [ 991.103494] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 991.137182] alg: No test for adler32 (adler32-zlib) [ 992.135678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 992.249970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 992.288176] LNet: Added LNI 192.168.201.16@tcp [8/256/0/180] [ 993.042131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 993.055904] LNet: 46343:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 993.059297] LNetError: 46343:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 993.069821] LNet: Removed LNI 192.168.201.16@tcp [ 993.084246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 993.148509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 993.162813] LNet: Added LNI 192.168.201.16@tcp [8/256/0/180] [ 993.459829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 993.482942] LNet: Removed LNI 192.168.201.16@tcp [ 993.498303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 993.555336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 993.579342] LNet: Added LNI 192.168.201.16@tcp [8/256/0/180] [ 993.876306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 993.893969] LNet: 46403:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 993.897929] LNet: 46403:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 993.902316] LNetError: 46403:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 993.907097] LNetError: 46403:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 993.916929] LNet: Removed LNI 192.168.201.16@tcp [ 993.932786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 994.008473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 994.076792] LNet: Added LNI 192.168.201.16@tcp [8/256/0/180] [ 994.429654] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 994.567095] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 994.572628] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 994.585479] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 996.982733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 997.014828] LNet: 46482:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 997.023967] LNetError: 46482:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 997.038795] LNet: Removed LNI 192.168.201.16@tcp [ 997.071870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 997.163933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 997.198970] LNet: Added LNI 192.168.201.16@tcp [8/256/0/180] [ 1005.330051] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 18:23:49 (1789597429) [ 1005.633321] LNet: 47073:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1005.660838] LNetError: 47073:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1005.718068] LNet: Removed LNI 192.168.201.16@tcp [ 1006.919607] Key type .llcrypt unregistered [ 1006.924836] Key type ._llcrypt unregistered [ 1017.324707] Key type ._llcrypt registered [ 1017.331742] Key type .llcrypt registered [ 1017.647567] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1017.692240] alg: No test for adler32 (adler32-zlib) [ 1018.850123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1018.947836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1018.980337] LNet: Added LNI 192.168.201.16@tcp [8/256/0/180] [ 1019.857572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1019.876466] LNet: 47476:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1019.881570] LNetError: 47476:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1019.900630] LNet: Removed LNI 192.168.201.16@tcp [ 1019.924685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1020.018468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 1020.069661] LNet: Added LNI 192.168.201.16@tcp [8/256/0/180] [ 1028.239430] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 18:24:12 (1789597452) [ 1028.330862] LNet: 48063:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1028.336355] LNetError: 48063:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1028.356444] LNet: Removed LNI 192.168.201.16@tcp [ 1029.265996] Key type .llcrypt unregistered [ 1029.270497] Key type ._llcrypt unregistered [ 1037.879573] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1038.045516] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1038.050921] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1038.053494] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1041.159603] Key type ._llcrypt registered [ 1041.161070] Key type .llcrypt registered [ 1041.499974] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1041.533062] alg: No test for adler32 (adler32-zlib) [ 1042.478380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1042.506659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1042.527847] LNet: Added LNI 192.168.201.16@tcp [8/256/0/180] [ 1043.701402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1043.728369] LNet: 48513:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1043.738725] LNetError: 48513:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1043.760186] LNet: Removed LNI 192.168.201.16@tcp [ 1043.797701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1043.908185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 1043.950209] LNet: Added LNI 192.168.201.16@tcp [8/256/0/180] [ 1052.969944] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 18:24:36 (1789597476) [ 1053.148727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1053.192505] LNet: 49105:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1053.208109] LNetError: 49105:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1053.244528] LNet: Removed LNI 192.168.201.16@tcp [ 1053.282571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1053.507527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 1061.522439] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 18:24:45 (1789597485) [ 1061.641388] LNet: 49687:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1062.365811] Key type .llcrypt unregistered [ 1062.368109] Key type ._llcrypt unregistered [ 1070.457502] Key type ._llcrypt registered [ 1070.460823] Key type .llcrypt registered [ 1070.743686] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1070.765862] alg: No test for adler32 (adler32-zlib) [ 1071.780484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1071.870892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1071.928195] LNet: Added LNI 192.168.201.16@tcp [8/256/0/180] [ 1073.365403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1073.382730] LNet: 50091:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1073.387793] LNetError: 50091:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1073.409230] LNet: Removed LNI 192.168.201.16@tcp [ 1073.446840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1073.520819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 1073.567302] LNet: Added LNI 192.168.201.16@tcp [8/256/0/180] [ 1081.927683] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 18:25:06 (1789597506) [ 1082.169754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1082.207132] LNet: 50680:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1082.216298] LNetError: 50680:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1082.248905] LNet: Removed LNI 192.168.201.16@tcp [ 1082.299251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1082.485456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 1089.895889] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 18:25:14 (1789597514) [ 1091.624316] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 1093.718122] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 18:25:18 (1789597518) [ 1095.439790] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 1098.389747] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 18:25:22 (1789597522) [ 1100.563273] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 1103.322355] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 18:25:27 (1789597527) [ 1105.217945] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 1107.951885] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 18:25:31 (1789597531) [ 1109.524733] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 1111.617495] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing ========================================================== 18:25:36 (1789597536) [ 1111.988483] LNet: 52710:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1112.839821] Key type .llcrypt unregistered [ 1112.847166] Key type ._llcrypt unregistered [ 1121.704056] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1121.909679] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1121.919940] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1121.927627] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1124.657940] Key type ._llcrypt registered [ 1124.666786] Key type .llcrypt registered [ 1124.953430] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1125.002053] alg: No test for adler32 (adler32-zlib) [ 1126.100087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1126.235595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1126.265098] LNet: Added LNI 192.168.201.16@tcp [8/512/0/180] [ 1126.272328] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 1126.278696] LNet: 53030:0:(api-ni.c:2991:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1126.422076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1126.451394] LNet: 53049:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1126.463629] LNetError: 53049:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1126.474697] LNet: Removed LNI 10.1.2.3@tcp [ 1126.478226] LNetError: 53049:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1126.492885] LNet: Removed LNI 192.168.201.16@tcp [ 1126.526232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1126.618369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1126.646328] LNet: Added LNI 192.168.201.16@tcp [8/256/0/180] [ 1126.793264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1126.841750] LNet: 53073:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1126.852321] LNetError: 53073:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1126.886098] LNet: Removed LNI 192.168.201.16@tcp [ 1126.911772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1127.050118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1127.148359] LNet: Added LNI 192.168.201.16@tcp [8/256/0/180] [ 1127.166746] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1127.176326] LNet: 53081:0:(api-ni.c:2991:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1127.421525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1127.451606] LNet: 53099:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1127.464806] LNetError: 53099:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1127.472293] LNet: Removed LNI 10.1.2.3@tcp [ 1127.476233] LNetError: 53099:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1127.500756] LNet: Removed LNI 192.168.201.16@tcp [ 1127.566066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1127.678992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1127.716248] LNet: Added LNI 192.168.201.16@tcp [8/256/0/180] [ 1127.721241] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1127.727232] LNet: 53107:0:(api-ni.c:2991:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1127.973545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1128.003860] LNet: 53133:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1128.017781] LNetError: 53133:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1128.026198] LNet: Removed LNI 10.1.2.3@tcp [ 1128.032359] LNetError: 53133:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1128.070540] LNet: Removed LNI 192.168.201.16@tcp [ 1128.138489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1128.220518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1128.264078] LNet: Added LNI 192.168.201.16@tcp [8/256/0/182] [ 1128.273162] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 1128.474043] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1128.509553] LNet: 53161:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1128.526088] LNetError: 53161:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1128.542566] LNet: Removed LNI 10.1.2.3@tcp [ 1128.546232] LNetError: 53161:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1128.566953] LNet: Removed LNI 192.168.201.16@tcp [ 1128.612503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1128.712292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1128.742282] LNet: Added LNI 192.168.201.16@tcp [8/256/0/182] [ 1128.750043] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 1128.752638] LNet: 53169:0:(api-ni.c:2991:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1128.811814] LNet: Removed LNI 10.1.2.3@tcp [ 1128.822174] LNet: Removed LNI 192.168.201.16@tcp [ 1129.404930] Key type .llcrypt unregistered [ 1129.407919] Key type ._llcrypt unregistered [ 1143.703955] Key type ._llcrypt registered [ 1143.705451] Key type .llcrypt registered [ 1145.507567] Lustre: DEBUG MARKER: == sanity-lnet test 180: FastReg map error should not leak descriptors ========================================================== 18:26:09 (1789597569) [ 1147.385492] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_180 Need o2ib NETTYPE [ 1149.225934] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 18:26:14 (1789597574) [ 1149.669207] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1149.683146] alg: No test for adler32 (adler32-zlib) [ 1150.703939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1150.771206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1150.802481] LNet: Added LNI 192.168.201.16@tcp [8/256/0/180] [ 1151.547824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.201.16 [ 1151.580468] LNet: 54432:0:(config.c:1740:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1151.594139] LNet: Removed LNI 192.168.201.16@tcp [ 1158.551415] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 18:26:23 (1789597583) [ 1158.825500] LNet: 55002:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1158.830798] LNetError: 55002:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1158.866780] LNet: Removed LNI 192.168.201.16@tcp [ 1159.508455] Key type .llcrypt unregistered [ 1159.511059] Key type ._llcrypt unregistered [ 1166.512510] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1166.741349] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1168.411713] Key type ._llcrypt registered [ 1168.413737] Key type .llcrypt registered [ 1168.628618] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1168.654725] alg: No test for adler32 (adler32-zlib) [ 1170.044656] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1177.476680] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 18:26:42 (1789597602) [ 1177.873285] LNet: 55905:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1177.878734] LNetError: 55905:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1177.897297] LNet: Removed LNI 10.1.2.3@tcp [ 1178.511383] Key type .llcrypt unregistered [ 1178.514933] Key type ._llcrypt unregistered [ 1187.363987] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1187.875177] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1187.881857] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1187.889555] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1191.088948] Key type ._llcrypt registered [ 1191.094210] Key type .llcrypt registered [ 1191.453751] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1191.491961] alg: No test for adler32 (adler32-zlib) [ 1192.450880] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1201.429202] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 18:27:06 (1789597626) [ 1204.032483] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 1206.601989] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 18:27:10 (1789597630) [ 1206.708174] LNet: 57081:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1206.726213] LNetError: 57081:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1206.751362] LNet: Removed LNI 10.1.2.3@tcp [ 1207.287405] Key type .llcrypt unregistered [ 1207.290394] Key type ._llcrypt unregistered [ 1215.063326] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1215.398920] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1215.404420] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1215.424754] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1217.055197] Key type ._llcrypt registered [ 1217.057868] Key type .llcrypt registered [ 1217.339080] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1217.371526] alg: No test for adler32 (adler32-zlib) [ 1218.268125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1218.436612] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1228.316665] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 18:27:33 (1789597653) [ 1228.948783] LNet: 58308:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1228.953895] LNetError: 58308:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1228.980569] LNet: Removed LNI 10.1.2.3@tcp [ 1229.784600] Key type .llcrypt unregistered [ 1229.786728] Key type ._llcrypt unregistered [ 1237.478605] Key type ._llcrypt registered [ 1237.480979] Key type .llcrypt registered [ 1237.753854] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1237.787350] alg: No test for adler32 (adler32-zlib) [ 1238.721143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1238.749728] LNet: Added LNI 192.168.201.16@tcp [8/256/0/180] [ 1247.831963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.116@tcp [ 1248.619400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.16@tcp -d 192.168.201.116@tcp -m GET -e local_interrupt -r 1 [ 1248.671656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.16@tcp -d 192.168.201.16@tcp -m GET -e local_interrupt -r 1 [ 1248.736748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.116@tcp [ 1248.778547] LNet: There was an unexpected network error while writing to 192.168.201.116: rc = -22 [ 1248.997552] LNet: 1 local NIs in recovery (showing 1): 192.168.201.16@tcp [ 1249.106389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1249.152433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1249.202569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1249.259348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1249.313952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1249.359277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1249.399963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1249.433204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1249.477542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1249.507483] LNet: 58745:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1249.533276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1249.620762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1249.677561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1249.721688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1249.779559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1249.861189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1249.915631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1250.621932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.16@tcp -d 192.168.201.116@tcp -m GET -e local_dropped -r 1 [ 1250.656244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.16@tcp -d 192.168.201.16@tcp -m GET -e local_dropped -r 1 [ 1250.722646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.116@tcp [ 1250.754140] LNet: There was an unexpected network error while writing to 192.168.201.116: rc = -22 [ 1251.164637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1251.208290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1251.251464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1251.303837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1251.344269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1251.399651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1251.461430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1251.493269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1251.557551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1251.586973] LNet: 58856:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1251.635484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1251.673423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1251.711548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1251.756401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1251.802071] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1251.844623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1251.889541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1252.661202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.16@tcp -d 192.168.201.116@tcp -m GET -e local_aborted -r 1 [ 1252.707960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.16@tcp -d 192.168.201.16@tcp -m GET -e local_aborted -r 1 [ 1252.784704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.116@tcp [ 1252.834869] LNet: There was an unexpected network error while writing to 192.168.201.116: rc = -22 [ 1252.852631] LNet: Skipped 1 previous similar message [ 1253.270234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1253.331626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1253.379332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1253.422787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1253.503387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1253.541649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1253.605113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1253.678584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1253.724706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1253.757935] LNet: 58968:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1253.800570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1253.879571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1253.955766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1254.039034] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1254.106338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1254.145703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1254.214462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1254.964063] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.16@tcp -d 192.168.201.116@tcp -m GET -e local_no_route -r 1 [ 1255.019996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.16@tcp -d 192.168.201.16@tcp -m GET -e local_no_route -r 1 [ 1255.063987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.116@tcp [ 1255.099897] LNet: There was an unexpected network error while writing to 192.168.201.116: rc = -22 [ 1255.113289] LNet: Skipped 1 previous similar message [ 1255.368682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1255.397825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1255.422308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1255.452869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1255.483675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1255.510439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1255.535966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1255.564463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1255.595343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1255.628347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1255.652961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1255.681972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1255.707819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1255.746335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1255.773249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1255.804243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1256.305790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.16@tcp -d 192.168.201.116@tcp -m GET -e local_timeout -r 1 [ 1256.335642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.16@tcp -d 192.168.201.16@tcp -m GET -e local_timeout -r 1 [ 1256.372747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.116@tcp [ 1256.605685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1256.647035] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1256.684888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1256.716923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1256.750202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1256.786504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1256.827134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1256.874332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1256.920466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1256.960808] LNet: 59190:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1256.966564] LNet: 59190:0:(api-ni.c:301:recovery_interval_set()) Skipped 1 previous similar message [ 1256.985977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1257.044165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1257.075169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1257.099223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1257.127584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1257.152882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1257.184514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1257.613452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.16@tcp -d 192.168.201.116@tcp -m GET -e local_error -r 1 [ 1257.640431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.16@tcp -d 192.168.201.16@tcp -m GET -e local_error -r 1 [ 1257.665808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.116@tcp [ 1257.924027] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1257.956555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1257.981940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1258.014575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1258.046958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1258.087483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1258.134119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1258.180311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1258.223623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1258.262601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1258.305779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1258.351825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1258.387713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1258.440767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1258.479549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1258.507847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1258.711915] LNet: 59343:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1258.717729] LNetError: 59343:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1259.751489] LNet: Removed LNI 192.168.201.16@tcp [ 1260.102796] Key type .llcrypt unregistered [ 1260.110331] Key type ._llcrypt unregistered [ 1282.375602] Key type ._llcrypt registered [ 1282.379423] Key type .llcrypt registered [ 1284.560947] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 18:28:28 (1789597708) [ 1285.485421] Key type .llcrypt unregistered [ 1285.487402] Key type ._llcrypt unregistered [ 1293.904396] Key type ._llcrypt registered [ 1293.919385] Key type .llcrypt registered [ 1294.421720] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1294.456612] alg: No test for adler32 (adler32-zlib) [ 1295.454404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1295.525958] LNet: Added LNI 192.168.201.16@tcp [8/256/0/180] [ 1304.234396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.116@tcp [ 1317.099743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1317.163543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1317.196344] LNet: Added LNI 192.168.201.16@tcp1 [8/256/0/180] [ 1319.372744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.16@tcp -d 192.168.201.116@tcp -m GET -e local_interrupt -r 1 [ 1319.443588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.16@tcp -d 192.168.201.116@tcp1 -m GET -e local_interrupt -r 1 [ 1319.508961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.16@tcp -d 192.168.201.16@tcp -m GET -e local_interrupt -r 1 [ 1319.597897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.16@tcp -d 192.168.201.16@tcp1 -m GET -e local_interrupt -r 1 [ 1319.636640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.16@tcp1 -d 192.168.201.116@tcp -m GET -e local_interrupt -r 1 [ 1319.673811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.16@tcp1 -d 192.168.201.116@tcp1 -m GET -e local_interrupt -r 1 [ 1319.702821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.16@tcp1 -d 192.168.201.16@tcp -m GET -e local_interrupt -r 1 [ 1319.727680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.16@tcp1 -d 192.168.201.16@tcp1 -m GET -e local_interrupt -r 1 [ 1319.757496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.116@tcp [ 1319.774205] LNet: There was an unexpected network error while writing to 192.168.201.116: rc = -22 [ 1319.779771] LNet: 1 local NIs in recovery (showing 1): 192.168.201.16@tcp [ 1319.954066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1319.975840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1319.997862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1320.023407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1320.052876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1320.074677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1320.099813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1320.125296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1320.164714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1320.179614] LNet: 60812:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1320.215308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1320.272426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1320.314449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1320.361320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1320.393367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1320.425378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1320.452471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1320.999392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.16@tcp -d 192.168.201.116@tcp -m GET -e local_dropped -r 1 [ 1321.053216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.16@tcp -d 192.168.201.116@tcp1 -m GET -e local_dropped -r 1 [ 1321.100652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.16@tcp -d 192.168.201.16@tcp -m GET -e local_dropped -r 1 [ 1321.146920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.16@tcp -d 192.168.201.16@tcp1 -m GET -e local_dropped -r 1 [ 1321.201343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.16@tcp1 -d 192.168.201.116@tcp -m GET -e local_dropped -r 1 [ 1321.262980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.16@tcp1 -d 192.168.201.116@tcp1 -m GET -e local_dropped -r 1 [ 1321.313955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.16@tcp1 -d 192.168.201.16@tcp -m GET -e local_dropped -r 1 [ 1321.352204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.16@tcp1 -d 192.168.201.16@tcp1 -m GET -e local_dropped -r 1 [ 1321.400376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.116@tcp [ 1321.442931] LNet: There was an unexpected network error while writing to 192.168.201.116: rc = -22 [ 1321.453236] LNet: Skipped 2 previous similar messages [ 1321.700094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1321.727743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1321.754893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1321.796355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1321.854163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1321.889231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1321.945414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1321.986929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1322.019456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1322.049067] LNet: 60935:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1322.078709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1322.130825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1322.175186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1322.208658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1322.236873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1322.267969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1322.297311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1322.813251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.16@tcp -d 192.168.201.116@tcp -m GET -e local_aborted -r 1 [ 1322.846275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.16@tcp -d 192.168.201.116@tcp1 -m GET -e local_aborted -r 1 [ 1322.879300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.16@tcp -d 192.168.201.16@tcp -m GET -e local_aborted -r 1 [ 1322.924923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.16@tcp -d 192.168.201.16@tcp1 -m GET -e local_aborted -r 1 [ 1322.975461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.16@tcp1 -d 192.168.201.116@tcp -m GET -e local_aborted -r 1 [ 1323.027898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.16@tcp1 -d 192.168.201.116@tcp1 -m GET -e local_aborted -r 1 [ 1323.051993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.16@tcp1 -d 192.168.201.16@tcp -m GET -e local_aborted -r 1 [ 1323.102712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.16@tcp1 -d 192.168.201.16@tcp1 -m GET -e local_aborted -r 1 [ 1323.160201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.116@tcp [ 1323.174562] LNet: There was an unexpected network error while writing to 192.168.201.116: rc = -22 [ 1323.179392] LNet: Skipped 2 previous similar messages [ 1323.438914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1323.503715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1323.573945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1323.631283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1323.680322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1323.730160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1323.758594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1323.780817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1323.816702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1323.837365] LNet: 61058:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1323.856958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1323.887934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1323.916389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1323.947382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1323.984933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1324.027319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1324.069443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1324.600853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.16@tcp -d 192.168.201.116@tcp -m GET -e local_no_route -r 1 [ 1324.636977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.16@tcp -d 192.168.201.116@tcp1 -m GET -e local_no_route -r 1 [ 1324.673389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.16@tcp -d 192.168.201.16@tcp -m GET -e local_no_route -r 1 [ 1324.713675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.16@tcp -d 192.168.201.16@tcp1 -m GET -e local_no_route -r 1 [ 1324.753081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.16@tcp1 -d 192.168.201.116@tcp -m GET -e local_no_route -r 1 [ 1324.800648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.16@tcp1 -d 192.168.201.116@tcp1 -m GET -e local_no_route -r 1 [ 1324.846043] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.16@tcp1 -d 192.168.201.16@tcp -m GET -e local_no_route -r 1 [ 1324.887661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.16@tcp1 -d 192.168.201.16@tcp1 -m GET -e local_no_route -r 1 [ 1324.937659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.116@tcp [ 1325.177340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1325.196257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1325.214950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1325.245416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1325.278718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1325.320327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1325.357671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1325.387327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1325.449078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1325.495484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1325.522372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1325.545133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1325.586030] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1325.630112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1325.666640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1325.703613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1326.270251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.16@tcp -d 192.168.201.116@tcp -m GET -e local_timeout -r 1 [ 1326.308950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.16@tcp -d 192.168.201.116@tcp1 -m GET -e local_timeout -r 1 [ 1326.343638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.16@tcp -d 192.168.201.16@tcp -m GET -e local_timeout -r 1 [ 1326.385165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.16@tcp -d 192.168.201.16@tcp1 -m GET -e local_timeout -r 1 [ 1326.429060] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.16@tcp1 -d 192.168.201.116@tcp -m GET -e local_timeout -r 1 [ 1326.461833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.16@tcp1 -d 192.168.201.116@tcp1 -m GET -e local_timeout -r 1 [ 1326.497127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.16@tcp1 -d 192.168.201.16@tcp -m GET -e local_timeout -r 1 [ 1326.536375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.16@tcp1 -d 192.168.201.16@tcp1 -m GET -e local_timeout -r 1 [ 1326.572275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.116@tcp [ 1326.591073] LNet: There was an unexpected network error while writing to 192.168.201.116: rc = -22 [ 1326.596558] LNet: Skipped 5 previous similar messages [ 1326.805507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1326.846376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1326.889959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1326.925993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1326.957344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1326.993295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1327.043671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1327.083753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1327.123573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1327.140954] LNet: 61305:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1327.147605] LNet: 61305:0:(api-ni.c:301:recovery_interval_set()) Skipped 1 previous similar message [ 1327.166377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1327.202380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1327.231767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1327.264495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1327.309727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1327.354979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1327.386283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1327.893296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.16@tcp -d 192.168.201.116@tcp -m GET -e local_error -r 1 [ 1327.924611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.16@tcp -d 192.168.201.116@tcp1 -m GET -e local_error -r 1 [ 1327.953239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.16@tcp -d 192.168.201.16@tcp -m GET -e local_error -r 1 [ 1327.980134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.16@tcp -d 192.168.201.16@tcp1 -m GET -e local_error -r 1 [ 1328.019908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.16@tcp1 -d 192.168.201.116@tcp -m GET -e local_error -r 1 [ 1328.048852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.16@tcp1 -d 192.168.201.116@tcp1 -m GET -e local_error -r 1 [ 1328.083843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.16@tcp1 -d 192.168.201.16@tcp -m GET -e local_error -r 1 [ 1328.119870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.16@tcp1 -d 192.168.201.16@tcp1 -m GET -e local_error -r 1 [ 1328.146230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.116@tcp [ 1328.350342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1328.385647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1328.420468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1328.448272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1328.472785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1328.500713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1328.528149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1328.566487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1328.594609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1328.628816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1328.659595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1328.688893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1328.720709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1328.750298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1328.794635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1328.827668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1335.327521] LNet: 61505:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1335.345235] LNetError: 61505:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1336.416481] LNet: Removed LNI 192.168.201.16@tcp [ 1336.420975] LNetError: 61505:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1336.443557] LNet: Removed LNI 192.168.201.16@tcp1 [ 1336.977874] Key type .llcrypt unregistered [ 1336.980366] Key type ._llcrypt unregistered [ 1355.472079] Key type ._llcrypt registered [ 1355.473184] Key type .llcrypt registered [ 1356.815890] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 18:29:41 (1789597781) [ 1357.959580] Key type .llcrypt unregistered [ 1357.964996] Key type ._llcrypt unregistered [ 1365.124710] Key type ._llcrypt registered [ 1365.127376] Key type .llcrypt registered [ 1365.488563] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1365.505381] alg: No test for adler32 (adler32-zlib) [ 1366.466613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1366.511290] LNet: Added LNI 192.168.201.16@tcp [8/256/0/180] [ 1375.748743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.116@tcp [ 1376.824827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.16@tcp -d 192.168.201.116@tcp -m GET -e remote_dropped -r 1 [ 1376.892866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.16@tcp -d 192.168.201.16@tcp -m GET -e remote_dropped -r 1 [ 1376.976224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.116@tcp [ 1376.999523] LNet: There was an unexpected network error while writing to 192.168.201.116: rc = -22 [ 1377.346775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1377.424309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1377.469378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1377.524626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1377.552318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1377.604489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1377.650824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1377.678532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1377.714694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1377.735719] LNet: 62752:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1377.776227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1377.833968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1377.861934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1377.910452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1377.951297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1377.977757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1378.032188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1378.783561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.16@tcp -d 192.168.201.116@tcp -m GET -e remote_error -r 1 [ 1378.852996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.16@tcp -d 192.168.201.16@tcp -m GET -e remote_error -r 1 [ 1378.922114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.116@tcp [ 1378.965216] LNet: There was an unexpected network error while writing to 192.168.201.116: rc = -22 [ 1379.299304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1379.346622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1379.392333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1379.436263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1379.512201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1379.562027] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1379.619355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1379.678181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1379.724043] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1379.750451] LNet: 62863:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1379.777376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1379.820289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1379.868651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1379.921521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1379.969249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1380.033403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1380.087477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1380.839210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.16@tcp -d 192.168.201.116@tcp -m GET -e remote_timeout -r 1 [ 1380.900657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.16@tcp -d 192.168.201.16@tcp -m GET -e remote_timeout -r 1 [ 1380.945384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.116@tcp [ 1380.971041] LNet: There was an unexpected network error while writing to 192.168.201.116: rc = -22 [ 1381.223743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1381.251501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1381.287843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1381.313725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1381.340345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1381.368428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1381.398686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1381.436858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1381.484540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1381.529690] LNet: 62975:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1381.552848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1381.583168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1381.609753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1381.664361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1381.741407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1381.800915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1381.846018] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1382.221715] LNet: 63016:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1382.234756] LNetError: 63016:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1383.271791] LNet: Removed LNI 192.168.201.16@tcp [ 1383.803654] Key type .llcrypt unregistered [ 1383.808869] Key type ._llcrypt unregistered [ 1404.032754] Key type ._llcrypt registered [ 1404.035853] Key type .llcrypt registered [ 1405.581060] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 18:30:29 (1789597829) [ 1406.703514] Key type .llcrypt unregistered [ 1406.710285] Key type ._llcrypt unregistered [ 1414.208680] Key type ._llcrypt registered [ 1414.210646] Key type .llcrypt registered [ 1414.444246] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1414.476679] alg: No test for adler32 (adler32-zlib) [ 1415.449440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1415.515359] LNet: Added LNI 192.168.201.16@tcp [8/256/0/180] [ 1423.960251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.116@tcp [ 1434.167375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1434.236692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1434.282988] LNet: Added LNI 192.168.201.16@tcp1 [8/256/0/180] [ 1436.392634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.16@tcp -d 192.168.201.116@tcp -m GET -e remote_dropped -r 1 [ 1436.440829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.16@tcp -d 192.168.201.116@tcp1 -m GET -e remote_dropped -r 1 [ 1436.485596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.16@tcp -d 192.168.201.16@tcp -m GET -e remote_dropped -r 1 [ 1436.541929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.16@tcp -d 192.168.201.16@tcp1 -m GET -e remote_dropped -r 1 [ 1436.593303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.16@tcp1 -d 192.168.201.116@tcp -m GET -e remote_dropped -r 1 [ 1436.646884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.16@tcp1 -d 192.168.201.116@tcp1 -m GET -e remote_dropped -r 1 [ 1436.689208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.16@tcp1 -d 192.168.201.16@tcp -m GET -e remote_dropped -r 1 [ 1436.762229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.16@tcp1 -d 192.168.201.16@tcp1 -m GET -e remote_dropped -r 1 [ 1436.845358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.116@tcp [ 1436.885749] LNet: There was an unexpected network error while writing to 192.168.201.116: rc = -22 [ 1436.889849] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.116@tcp [ 1437.202947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1437.240348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1437.294732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1437.322420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1437.353855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1437.386719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1437.437971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1437.468423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1437.520460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1437.546987] LNet: 64484:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1437.568295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1437.622199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1437.676304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1437.737492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1437.793377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1437.871070] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1437.946886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1438.047757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1438.807406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.16@tcp -d 192.168.201.116@tcp -m GET -e remote_error -r 1 [ 1438.847263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.16@tcp -d 192.168.201.116@tcp1 -m GET -e remote_error -r 1 [ 1438.895279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.16@tcp -d 192.168.201.16@tcp -m GET -e remote_error -r 1 [ 1438.951577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.16@tcp -d 192.168.201.16@tcp1 -m GET -e remote_error -r 1 [ 1439.010701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.16@tcp1 -d 192.168.201.116@tcp -m GET -e remote_error -r 1 [ 1439.072271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.16@tcp1 -d 192.168.201.116@tcp1 -m GET -e remote_error -r 1 [ 1439.138927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.16@tcp1 -d 192.168.201.16@tcp -m GET -e remote_error -r 1 [ 1439.203248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.16@tcp1 -d 192.168.201.16@tcp1 -m GET -e remote_error -r 1 [ 1439.268339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.116@tcp [ 1439.303333] LNet: There was an unexpected network error while writing to 192.168.201.116: rc = -22 [ 1439.311854] LNet: Skipped 2 previous similar messages [ 1439.597538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1439.649479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1439.711363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1439.764453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1439.851833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1439.897700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1439.979265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1440.023480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1440.066897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1440.088263] LNet: 64609:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1440.121405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1440.162938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1440.214216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1440.249269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1440.296144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1440.370883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1440.463986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1440.534964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1441.386501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.16@tcp -d 192.168.201.116@tcp -m GET -e remote_timeout -r 1 [ 1441.482285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.16@tcp -d 192.168.201.116@tcp1 -m GET -e remote_timeout -r 1 [ 1441.577848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.16@tcp -d 192.168.201.16@tcp -m GET -e remote_timeout -r 1 [ 1441.684904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.16@tcp -d 192.168.201.16@tcp1 -m GET -e remote_timeout -r 1 [ 1441.766304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.16@tcp1 -d 192.168.201.116@tcp -m GET -e remote_timeout -r 1 [ 1441.883273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.16@tcp1 -d 192.168.201.116@tcp1 -m GET -e remote_timeout -r 1 [ 1441.955601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.16@tcp1 -d 192.168.201.16@tcp -m GET -e remote_timeout -r 1 [ 1442.009901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.16@tcp1 -d 192.168.201.16@tcp1 -m GET -e remote_timeout -r 1 [ 1442.085030] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.116@tcp [ 1442.128438] LNet: There was an unexpected network error while writing to 192.168.201.116: rc = -22 [ 1442.642867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1442.682560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1442.769194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1442.834454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1442.893131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1442.964744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1443.015431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1443.078744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1443.146872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1443.183973] LNet: 64735:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1443.231618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1443.300149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1443.361220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1443.440718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1443.527620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1443.559930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1443.619943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1443.685588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1449.352815] LNet: 64813:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1449.358799] LNetError: 64813:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1450.400218] LNet: Removed LNI 192.168.201.16@tcp [ 1450.403189] LNetError: 64813:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1450.414544] LNet: Removed LNI 192.168.201.16@tcp1 [ 1451.018114] Key type .llcrypt unregistered [ 1451.020308] Key type ._llcrypt unregistered [ 1472.774630] Key type ._llcrypt registered [ 1472.779944] Key type .llcrypt registered [ 1474.699300] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 18:31:38 (1789597898) [ 1476.759858] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 1479.213420] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 18:31:43 (1789597903) [ 1480.142545] Key type .llcrypt unregistered [ 1480.144157] Key type ._llcrypt unregistered [ 1487.337428] Key type ._llcrypt registered [ 1487.342157] Key type .llcrypt registered [ 1487.530976] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1487.549633] alg: No test for adler32 (adler32-zlib) [ 1488.649724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1488.674209] LNet: Added LNI 192.168.201.16@tcp [8/256/0/180] [ 1497.605281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.116@tcp [ 1498.590560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.16@tcp -d 192.168.201.116@tcp -m GET -e network_timeout -r 1 [ 1498.681404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.16@tcp -d 192.168.201.16@tcp -m GET -e network_timeout -r 1 [ 1498.750921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.116@tcp [ 1498.785854] LNet: There was an unexpected network error while writing to 192.168.201.116: rc = -22 [ 1498.924376] LNet: 1 local NIs in recovery (showing 1): 192.168.201.16@tcp [ 1499.185418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1499.242684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1499.339730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1499.401351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1499.451678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1499.504861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1499.599923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1499.699687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1499.760344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1499.788557] LNet: 66357:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1499.815892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1499.856779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1499.933931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1500.004065] LNet: There was an unexpected network error while writing to 192.168.201.16: rc = -22 [ 1500.009886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1500.048952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1500.126124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1500.615361] LNet: 66396:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1500.623371] LNetError: 66396:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1500.650198] LNet: Removed LNI 192.168.201.16@tcp [ 1501.252528] Key type .llcrypt unregistered [ 1501.256908] Key type ._llcrypt unregistered [ 1526.365348] Key type ._llcrypt registered [ 1526.374980] Key type .llcrypt registered [ 1526.597082] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1526.618926] alg: No test for adler32 (adler32-zlib) [ 1527.635357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1527.680503] LNet: Added LNI 192.168.201.16@tcp [8/256/0/180] [ 1536.465991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.116@tcp [ 1546.355870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1546.425234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1546.460233] LNet: Added LNI 192.168.201.16@tcp1 [8/256/0/180] [ 1547.966435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.16@tcp -d 192.168.201.116@tcp -m GET -e network_timeout -r 1 [ 1548.036369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.16@tcp -d 192.168.201.116@tcp1 -m GET -e network_timeout -r 1 [ 1548.115100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.16@tcp -d 192.168.201.16@tcp -m GET -e network_timeout -r 1 [ 1548.169942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.16@tcp -d 192.168.201.16@tcp1 -m GET -e network_timeout -r 1 [ 1548.197485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.16@tcp1 -d 192.168.201.116@tcp -m GET -e network_timeout -r 1 [ 1548.221352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.16@tcp1 -d 192.168.201.116@tcp1 -m GET -e network_timeout -r 1 [ 1548.256978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.16@tcp1 -d 192.168.201.16@tcp -m GET -e network_timeout -r 1 [ 1548.325930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.16@tcp1 -d 192.168.201.16@tcp1 -m GET -e network_timeout -r 1 [ 1548.379856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.116@tcp [ 1548.416778] LNet: There was an unexpected network error while writing to 192.168.201.116: rc = -22 [ 1548.738047] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1548.798178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1548.854491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1548.897378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1548.932309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1548.971689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1549.025697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1549.083818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1549.128274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1549.153537] LNet: 67276:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1549.182879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1549.213617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1549.216539] LNet: 1 local NIs in recovery (showing 1): 192.168.201.16@tcp [ 1549.218108] LNet: There was an unexpected network error while writing to 192.168.201.116: rc = -22 [ 1549.246216] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.116@tcp [ 1549.266632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1549.304917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1549.371328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1549.427685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1555.182353] LNet: There was an unexpected network error while writing to 192.168.201.116: rc = -22 [ 1555.185295] LNet: Skipped 1 previous similar message [ 1555.555652] LNet: 67350:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1555.566535] LNetError: 67350:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1556.576186] LNet: Removed LNI 192.168.201.16@tcp [ 1556.578240] LNetError: 67350:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1556.595074] LNet: Removed LNI 192.168.201.16@tcp1 [ 1557.030665] Key type .llcrypt unregistered [ 1557.035618] Key type ._llcrypt unregistered [ 1573.568507] Key type ._llcrypt registered [ 1573.570759] Key type .llcrypt registered [ 1575.060063] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 18:33:19 (1789597999) [ 1575.482608] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1575.497437] alg: No test for adler32 (adler32-zlib) [ 1576.491769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1576.559993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1576.593651] LNet: Added LNI 192.168.201.16@tcp [8/256/0/180] [ 1577.647293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1577.682431] LNet: Added LNI 192.168.201.16@tcp1 [8/256/0/180] [ 1578.775611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.16@tcp [ 1578.885819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1578.984569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1579.028956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1579.068192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1579.158650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1579.214715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.201.16@tcp [ 1579.238884] LNetError: 68446:0:(api-ni.c:4478:lnet_ni_set_healthv()) manually adding local NI 192.168.201.16@tcp to recovery [ 1579.617710] LNet: 1 local NIs in recovery (showing 1): 192.168.201.16@tcp [ 1579.618286] LNet: There was an unexpected network error while writing to 192.168.201.16: rc = -22 [ 1581.666139] LNet: There was an unexpected network error while writing to 192.168.201.16: rc = -22 [ 1582.889972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1585.760908] LNet: There was an unexpected network error while writing to 192.168.201.16: rc = -22 [ 1586.675277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1586.723094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1586.813383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1586.859523] LNet: 68553:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1586.866507] LNetError: 68553:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1587.872987] LNet: Removed LNI 192.168.201.16@tcp [ 1587.874658] LNetError: 68553:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1587.905335] LNet: Removed LNI 192.168.201.16@tcp1 [ 1587.939602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1588.056366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1588.077304] LNet: Added LNI 192.168.201.16@tcp [8/256/0/180] [ 1589.202320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1589.223495] LNet: Added LNI 192.168.201.16@tcp1 [8/256/0/180] [ 1590.067345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.16@tcp [ 1590.134901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1590.220518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1590.294750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1590.337441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1590.390323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1590.436487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1590.486518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.201.16@tcp [ 1590.502642] LNetError: 68826:0:(api-ni.c:4478:lnet_ni_set_healthv()) manually adding local NI 192.168.201.16@tcp to recovery [ 1591.072590] LNet: There was an unexpected network error while writing to 192.168.201.16: rc = -22 [ 1594.238479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1597.217992] LNet: There was an unexpected network error while writing to 192.168.201.16: rc = -22 [ 1597.234790] LNet: Skipped 1 previous similar message [ 1602.346209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1602.393868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1602.449611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1610.841178] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 18:33:55 (1789598035) [ 1611.064825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1611.089967] LNet: 69544:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1611.095648] LNetError: 69544:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1611.107451] LNet: Removed LNI 192.168.201.16@tcp [ 1611.126714] LNet: Removed LNI 192.168.201.16@tcp1 [ 1611.144239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1611.193426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1611.215114] LNet: Added LNI 192.168.201.16@tcp [8/256/0/180] [ 1611.894781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1611.912289] LNet: Added LNI 192.168.201.16@tcp1 [8/256/0/180] [ 1612.947181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.16@tcp [ 1613.124985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1613.163913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1613.240129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1613.334993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1613.401639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1613.444733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.201.16@tcp --health 0 [ 1613.626065] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1614.240488] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.16@tcp [ 1614.244757] LNet: There was an unexpected network error while writing to 192.168.201.16: rc = -22 [ 1614.272768] LNet: Skipped 1 previous similar message [ 1629.447759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1629.739337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1629.792325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1629.838650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.201.16@tcp --health 500 [ 1630.016223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1636.863556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1636.917646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1636.936698] LNet: 70036:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1636.940741] LNetError: 70036:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1636.947234] LNetError: 70036:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1636.959826] LNet: Removed LNI 192.168.201.16@tcp [ 1636.974126] LNet: Removed LNI 192.168.201.16@tcp1 [ 1636.994478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1637.083789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1637.110927] LNet: Added LNI 192.168.201.16@tcp [8/256/0/180] [ 1638.047813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1638.069724] LNet: Added LNI 192.168.201.16@tcp1 [8/256/0/180] [ 1638.812966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.16@tcp [ 1638.837621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1638.887374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1638.919339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1638.954798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1638.984469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1639.003223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1639.019592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.201.16@tcp --health 0 [ 1640.111189] LNet: There was an unexpected network error while writing to 192.168.201.16: rc = -22 [ 1640.121665] LNet: Skipped 3 previous similar messages [ 1640.384323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1650.722443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1650.749816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1650.799967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1658.747924] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 18:34:43 (1789598083) [ 1659.173648] LNet: 71083:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1659.178203] LNetError: 71083:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1659.185303] LNetError: 71083:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1659.192769] LNet: Removed LNI 192.168.201.16@tcp [ 1659.221507] LNet: Removed LNI 192.168.201.16@tcp1 [ 1659.787362] Key type .llcrypt unregistered [ 1659.790646] Key type ._llcrypt unregistered [ 1668.620471] Key type ._llcrypt registered [ 1668.624754] Key type .llcrypt registered [ 1669.016127] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1669.045546] alg: No test for adler32 (adler32-zlib) [ 1670.002549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1670.062807] LNet: Added LNI 192.168.201.16@tcp [8/256/0/180] [ 1680.926412] Lustre: DEBUG MARKER: Initial discovery [ 1681.337855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.201.116@tcp [ 1683.448577] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1683.811382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1683.851691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.201.116@tcp [ 1683.897902] LNet: There was an unexpected network error while writing to 192.168.201.116: rc = -22 [ 1683.917697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1683.966459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.201.116@tcp -r 1 -m GET -l 3 [ 1684.022717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.201.116@tcp -r 1 -m GET -e local_error [ 1684.074369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.201.116@tcp -r 1 -m PUT -l 6 [ 1684.449576] LNet: 1 local NIs in recovery (showing 1): 192.168.201.16@tcp [ 1685.551752] Lustre: DEBUG MARKER: Force oleg116-client.virtnet to discover 192.168.201.116@tcp (in background) [ 1685.829135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.201.116@tcp [ 1687.223633] Lustre: DEBUG MARKER: Force oleg116-server to discover 192.168.201.16@tcp [ 1688.887644] LNet: There was an unexpected network error while writing to 192.168.201.116: rc = -22 [ 1690.701782] Lustre: DEBUG MARKER: Wait for 71844 [ 1692.556557] Lustre: DEBUG MARKER: Finished wait on 71844 [ 1692.617300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay del -a [ 1692.675594] LNet: There was an unexpected network error while writing to 192.168.201.116: rc = -22 [ 1692.690703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1692.996783] LNet: 72292:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1693.002596] LNetError: 72292:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1693.029306] LNet: Removed LNI 192.168.201.16@tcp [ 1693.512452] Key type .llcrypt unregistered [ 1693.516613] Key type ._llcrypt unregistered [ 1710.754440] Key type ._llcrypt registered [ 1710.758515] Key type .llcrypt registered [ 1712.512866] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 18:35:36 (1789598136) [ 1713.060550] Key type .llcrypt unregistered [ 1713.063930] Key type ._llcrypt unregistered [ 1719.534250] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1719.780761] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1719.795195] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1719.800772] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1722.339223] Key type ._llcrypt registered [ 1722.341721] Key type .llcrypt registered [ 1722.543464] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1722.555647] alg: No test for adler32 (adler32-zlib) [ 1723.461342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1723.506362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1723.524891] LNet: Added LNI 192.168.201.16@tcp [8/256/0/180] [ 1724.379855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1724.399328] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1732.462953] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 18:35:57 (1789598157) [ 1732.823397] LNet: 74366:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1732.833101] LNetError: 74366:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1732.842535] LNet: Removed LNI 10.1.2.3@tcp [ 1732.857624] LNet: Removed LNI 192.168.201.16@tcp [ 1733.561532] Key type .llcrypt unregistered [ 1733.563609] Key type ._llcrypt unregistered [ 1741.200955] Key type ._llcrypt registered [ 1741.210353] Key type .llcrypt registered [ 1741.470793] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1741.500572] alg: No test for adler32 (adler32-zlib) [ 1742.438909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1742.492433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1742.518521] LNet: Added LNI 192.168.201.16@tcp [8/256/0/180] [ 1743.203216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1743.222724] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1744.340848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.201.16@tcp [ 1745.377202] LNetError: Unexpected error -22 connecting to 192.168.201.16@tcp at host 192.168.201.16:988 [ 1753.042485] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 18:36:17 (1789598177) [ 1753.251981] LNet: 75491:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1753.270849] LNetError: 75491:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1753.277996] LNet: Removed LNI 10.1.2.3@tcp [ 1753.288919] LNet: Removed LNI 192.168.201.16@tcp [ 1754.068243] Key type .llcrypt unregistered [ 1754.071257] Key type ._llcrypt unregistered [ 1761.718299] Key type ._llcrypt registered [ 1761.720699] Key type .llcrypt registered [ 1762.083437] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1762.116204] alg: No test for adler32 (adler32-zlib) [ 1763.220778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1763.304574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1763.342388] LNet: Added LNI 192.168.201.16@tcp1 [8/256/0/180] [ 1764.189909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1764.206114] LNet: Added LNI 192.168.201.16@tcp2 [8/256/0/180] [ 1765.018971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.201.16@tcp1 --nid 192.168.201.16@tcp2 [ 1768.684625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.16@tcp1 192.168.201.16@tcp2 [ 1768.699106] LNetError: 76326:0:(lib-move.c:2094:lnet_handle_find_routed_path()) Unable to send message from 192.168.201.16@tcp1 to 192.168.201.16@tcp2 - Route table may be misconfigured [ 1768.715292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.16@tcp2 192.168.201.16@tcp1 [ 1775.104643] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 18:36:39 (1789598199) [ 1775.477437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1775.504633] LNet: 76900:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1775.510868] LNetError: 76900:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1776.544307] LNet: Removed LNI 192.168.201.16@tcp1 [ 1776.549526] LNetError: 76900:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1776.581830] LNet: Removed LNI 192.168.201.16@tcp2 [ 1776.616987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1776.663621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1776.682749] LNet: Added LNI 192.168.201.16@tcp [8/256/0/180] [ 1777.270830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1777.287283] LNet: Added LNI 192.168.201.16@tcp1 [8/256/0/180] [ 1778.136364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.16@tcp [ 1778.171909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.16@tcp 192.168.201.16@tcp [ 1778.218673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.16@tcp1 192.168.201.16@tcp1 [ 1778.255764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.201.16@tcp -d 192.168.201.16@tcp -e network_timeout [ 1778.288783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.201.16@tcp -d 192.168.201.16@tcp1 -e network_timeout [ 1778.337099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.201.16@tcp1 -d 192.168.201.16@tcp -e network_timeout [ 1778.374614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.201.16@tcp1 -d 192.168.201.16@tcp1 -e network_timeout [ 1778.409431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.16@tcp [ 1778.430472] LNet: There was an unexpected network error while writing to 192.168.201.16: rc = -22 [ 1778.443234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1778.531387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1778.606120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1778.723639] LNet: 1 local NIs in recovery (showing 1): 192.168.201.16@tcp [ 1784.934711] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 18:36:49 (1789598209) [ 1785.095504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1785.111467] LNet: 77761:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1785.116304] LNetError: 77761:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1786.144519] LNet: Removed LNI 192.168.201.16@tcp [ 1787.177580] LNet: Removed LNI 192.168.201.16@tcp1 [ 1787.228661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1787.356247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1787.700963] LNet: 77799:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1788.049721] Key type .llcrypt unregistered [ 1788.051967] Key type ._llcrypt unregistered [ 1800.178875] Key type ._llcrypt registered [ 1800.180625] Key type .llcrypt registered [ 1801.395865] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 18:37:06 (1789598226) [ 1801.929288] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1801.960871] alg: No test for adler32 (adler32-zlib) [ 1802.919152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1804.125164] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1805.737476] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 18:37:10 (1789598230) [ 1806.060476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1806.098752] LNet: 78895:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1806.135781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1806.270342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1806.295089] LNet: Added LNI 192.168.201.16@tcp [8/256/0/180] [ 1806.894431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1806.913554] LNet: Added LNI 192.168.201.16@tcp1 [8/256/0/180] [ 1807.897769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.16@tcp [ 1807.928775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.16@tcp1 [ 1807.964358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.16@tcp1 [ 1814.207530] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 18:37:19 (1789598239) [ 1815.878483] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg116-server [ 1817.861031] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 18:37:22 (1789598242) [ 1819.364948] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg116-server [ 1821.008349] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 18:37:26 (1789598246) [ 1822.665936] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg116-server [ 1824.579808] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 18:37:29 (1789598249) [ 1826.466567] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg116-server [ 1828.392593] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 18:37:33 (1789598253) [ 1829.903929] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg116-server [ 1831.456693] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 18:37:36 (1789598256) [ 1833.238485] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg116-server [ 1834.906450] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 18:37:39 (1789598259) [ 1836.846181] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg116-server [ 1838.886526] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 18:37:43 (1789598263) [ 1840.472809] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg116-server [ 1842.874610] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 18:37:47 (1789598267) [ 1844.597433] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg116-server [ 1846.739222] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 18:37:51 (1789598271) [ 1846.897238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1846.925662] LNet: 82769:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1846.939532] LNetError: 82769:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1846.949344] LNet: Removed LNI 192.168.201.16@tcp [ 1847.978670] LNet: Removed LNI 192.168.201.16@tcp1 [ 1848.009326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1848.087329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1848.122728] LNet: Added LNI 192.168.201.16@tcp [8/256/0/180] [ 1849.068298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1849.171904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.16@tcp [ 1849.300798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1849.318870] LNet: 82920:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1849.322508] LNetError: 82920:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1849.326721] LNetError: 82920:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1849.347492] LNet: Removed LNI 192.168.201.16@tcp [ 1849.385770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1849.473564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1849.508947] LNet: Added LNI 192.168.201.16@tcp [8/256/0/180] [ 1850.906442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.16@tcp [ 1851.177363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1851.211931] LNetError: 83065:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1852.262501] LNet: Removed LNI 192.168.201.16@tcp [ 1852.332668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1852.490485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1852.534302] LNet: Added LNI 192.168.201.16@tcp [8/256/0/180] [ 1853.613512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1853.746516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.16@tcp [ 1854.146785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1854.180585] LNet: 83213:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1854.188229] LNet: 83213:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1854.197073] LNetError: 83213:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1854.240182] LNet: Removed LNI 192.168.201.16@tcp [ 1854.297963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1854.470826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1854.555220] LNet: Added LNI 192.168.201.16@tcp [8/256/0/180] [ 1855.800065] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.16@tcp [ 1856.065626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1857.136800] LNet: Removed LNI 192.168.201.16@tcp [ 1857.175464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1857.253182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1857.267111] LNet: Added LNI 192.168.201.16@tcp [8/256/0/180] [ 1858.081577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1858.161309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.16@tcp [ 1858.411482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1858.431592] LNet: 83506:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1858.441857] LNet: 83506:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1858.455654] LNetError: 83506:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1858.471298] LNetError: 83506:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1858.493932] LNet: Removed LNI 192.168.201.16@tcp [ 1858.532156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1858.596730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1858.636420] LNet: Added LNI 192.168.201.16@tcp [8/256/0/180] [ 1859.544078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.16@tcp [ 1859.831389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1860.903966] LNet: Removed LNI 192.168.201.16@tcp [ 1860.931952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1861.049469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1861.100062] LNet: Added LNI 192.168.201.16@tcp [8/256/0/180] [ 1862.057442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1862.123664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.16@tcp [ 1862.361983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1863.401568] LNet: Removed LNI 192.168.201.16@tcp [ 1863.448952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1863.531550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1863.556139] LNet: Added LNI 192.168.201.16@tcp [8/256/0/180] [ 1864.395565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.16@tcp [ 1865.881659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1866.996136] LNet: Removed LNI 192.168.201.16@tcp [ 1867.040989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1867.168920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1867.206673] LNet: Added LNI 192.168.201.16@tcp [8/256/0/180] [ 1868.594329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1868.758987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.16@tcp [ 1869.122411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1869.164170] LNet: 84101:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1869.169629] LNet: 84101:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 1869.177919] LNetError: 84101:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1869.184602] LNetError: 84101:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 1869.198977] LNet: Removed LNI 192.168.201.16@tcp [ 1869.223074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1869.358247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1869.428084] LNet: Added LNI 192.168.201.16@tcp [8/256/0/180] [ 1870.530531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.16@tcp [ 1871.891559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1873.000367] LNet: Removed LNI 192.168.201.16@tcp [ 1873.017471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1873.149774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1873.177211] LNet: Added LNI 192.168.201.16@tcp [8/256/0/180] [ 1874.464217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1874.624601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.16@tcp [ 1876.356704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1877.425343] LNet: Removed LNI 192.168.201.16@tcp [ 1877.489853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1877.589917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1877.619273] LNet: Added LNI 192.168.201.16@tcp [8/256/0/180] [ 1878.332707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.16@tcp [ 1878.583647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1879.654791] LNet: Removed LNI 192.168.201.16@tcp [ 1879.670274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1879.718892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1879.740905] LNet: Added LNI 192.168.201.16@tcp [8/256/0/180] [ 1880.447168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1880.506971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1880.540131] LNet: Removed LNI 192.168.201.16@tcp [ 1880.570182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1880.650107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1880.681675] LNet: Added LNI 192.168.201.16@tcp [8/256/0/180] [ 1881.529199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1881.611742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1881.664992] LNet: Removed LNI 192.168.201.16@tcp [ 1881.687602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1881.712764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 128 [ 1881.776970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1881.816344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1881.913140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 1000000000000000000000000 [ 1882.002309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1882.037293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1882.094377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer foo [ 1882.163736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1882.199755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1882.243186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 8 [ 1882.263711] LNet: Added LNI 192.168.201.16@tcp [8/256/0/180] [ 1883.035972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1883.084994] LNet: Removed LNI 192.168.201.16@tcp [ 1883.108350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1883.185963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1889.899485] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 18:38:34 (1789598314) [ 1890.028971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1890.057911] LNet: 85574:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1890.064741] LNet: 85574:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 9 previous similar messages [ 1890.093626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1890.161654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1890.178746] LNet: Added LNI 192.168.201.16@tcp231 [8/256/0/180] [ 1891.190542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1891.208927] LNetError: 85717:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1891.214928] LNetError: 85717:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 6 previous similar messages [ 1891.240309] LNet: Removed LNI 192.168.201.16@tcp231 [ 1891.280272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1891.356543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1891.413850] LNet: Added LNI 192.168.201.16@tcp231 [8/256/0/180] [ 1891.423896] LNet: 85725:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1891.962780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1891.992864] LNet: Removed LNI 192.168.201.16@tcp231 [ 1892.023475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1892.046687] LNet: Added LNI 192.168.201.16@tcp231 [8/256/0/180] [ 1893.232975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1893.264706] LNet: Removed LNI 192.168.201.16@tcp231 [ 1893.282130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1893.330341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1893.356223] LNet: Added LNI 192.168.201.16@tcp231 [8/256/0/0] [ 1893.362716] LNet: 85877:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1893.828396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1893.867308] LNet: Removed LNI 192.168.201.16@tcp231 [ 1893.891201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1893.972286] LNet: Added LNI 192.168.201.16@tcp231 [8/256/0/0] [ 1895.089960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1895.142460] LNet: Removed LNI 192.168.201.16@tcp231 [ 1895.170444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1895.313169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1895.351483] LNet: Added LNI 192.168.201.16@tcp231 [8/256/0/60] [ 1895.358666] LNet: 86028:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1895.800931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1895.856906] LNet: Removed LNI 192.168.201.16@tcp231 [ 1895.880213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1895.921196] LNet: Added LNI 192.168.201.16@tcp231 [8/256/0/60] [ 1896.915696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1896.952100] LNet: Removed LNI 192.168.201.16@tcp231 [ 1896.984863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1897.064943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1897.091159] LNet: Added LNI 192.168.201.16@tcp231 [8/256/0/180] [ 1897.361126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1897.386164] LNet: Removed LNI 192.168.201.16@tcp231 [ 1897.398541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1897.424301] LNet: Added LNI 192.168.201.16@tcp231 [8/256/0/180] [ 1905.402673] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 18:38:49 (1789598329) [ 1905.690648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1905.755758] LNet: Removed LNI 192.168.201.16@tcp231 [ 1905.784791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1905.866792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1905.888914] LNet: Added LNI 192.168.201.16@tcp232 [8/256/0/180] [ 1906.490306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1906.528785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1906.559772] LNet: Removed LNI 192.168.201.16@tcp232 [ 1906.602465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1906.636357] LNet: Added LNI 192.168.201.16@tcp232 [8/256/0/180] [ 1907.319859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1907.354443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1907.388286] LNet: Removed LNI 192.168.201.16@tcp232 [ 1907.401499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1907.432691] LNet: Added LNI 192.168.201.16@tcp232 [8/256/0/180] [ 1908.382155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1908.417639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1908.438251] LNet: Removed LNI 192.168.201.16@tcp232 [ 1908.453772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1908.476575] LNet: Added LNI 192.168.201.16@tcp232 [8/256/0/180] [ 1909.116235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1909.151734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1909.193104] LNet: Removed LNI 192.168.201.16@tcp232 [ 1909.202951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1917.176036] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 18:39:01 (1789598341) [ 1918.956097] Key type .llcrypt unregistered [ 1918.961871] Key type ._llcrypt unregistered [ 1927.452083] Key type ._llcrypt registered [ 1927.456769] Key type .llcrypt registered [ 1927.678368] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1927.685515] alg: No test for adler32 (adler32-zlib) [ 1928.531161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1928.546064] LNet: Added LNI 192.168.201.16@tcp [8/256/0/180] [ 1937.204124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.116@tcp [ 1946.791580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1946.852525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1946.869436] LNet: Added LNI 192.168.201.16@tcp1 [8/256/0/180] [ 1948.367704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.16@tcp -d 192.168.201.116@tcp -m GET -e local_interrupt -r 2 [ 1948.405671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.16@tcp -d 192.168.201.116@tcp1 -m GET -e local_interrupt -r 2 [ 1948.441057] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.16@tcp -d 192.168.201.16@tcp -m GET -e local_interrupt -r 2 [ 1948.479279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.16@tcp -d 192.168.201.16@tcp1 -m GET -e local_interrupt -r 2 [ 1948.514706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.16@tcp1 -d 192.168.201.116@tcp -m GET -e local_interrupt -r 2 [ 1948.554787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.16@tcp1 -d 192.168.201.116@tcp1 -m GET -e local_interrupt -r 2 [ 1948.607192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.16@tcp1 -d 192.168.201.16@tcp -m GET -e local_interrupt -r 2 [ 1948.651968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.16@tcp1 -d 192.168.201.16@tcp1 -m GET -e local_interrupt -r 2 [ 1948.713669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.16@tcp 192.168.201.116@tcp [ 1948.733607] LNet: There was an unexpected network error while writing to 192.168.201.116: rc = -22 [ 1948.738688] LNet: 1 local NIs in recovery (showing 1): 192.168.201.16@tcp [ 1948.990159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1949.027423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1949.064115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1949.087922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1949.110682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1949.133545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1949.156805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1949.180434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1949.202583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1949.226719] LNet: 88682:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1949.264810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1949.314417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1949.380719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1949.437029] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1949.464529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1949.500346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1950.052722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.16@tcp -d 192.168.201.116@tcp -m GET -e local_dropped -r 2 [ 1950.085711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.16@tcp -d 192.168.201.116@tcp1 -m GET -e local_dropped -r 2 [ 1950.118775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.16@tcp -d 192.168.201.16@tcp -m GET -e local_dropped -r 2 [ 1950.162461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.16@tcp -d 192.168.201.16@tcp1 -m GET -e local_dropped -r 2 [ 1950.202968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.16@tcp1 -d 192.168.201.116@tcp -m GET -e local_dropped -r 2 [ 1950.238922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.16@tcp1 -d 192.168.201.116@tcp1 -m GET -e local_dropped -r 2 [ 1950.275508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.16@tcp1 -d 192.168.201.16@tcp -m GET -e local_dropped -r 2 [ 1950.311389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.16@tcp1 -d 192.168.201.16@tcp1 -m GET -e local_dropped -r 2 [ 1950.351637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.16@tcp 192.168.201.116@tcp [ 1950.370932] LNet: There was an unexpected network error while writing to 192.168.201.116: rc = -22 [ 1950.673409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1950.707437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1950.742516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1950.784117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1950.847170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1950.876694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1950.922668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1950.959738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1950.995651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1951.025528] LNet: 88804:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1951.061564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1951.109905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1951.148364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1951.171483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1951.192924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1951.230270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1951.740565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.16@tcp -d 192.168.201.116@tcp -m GET -e local_aborted -r 2 [ 1951.774581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.16@tcp -d 192.168.201.116@tcp1 -m GET -e local_aborted -r 2 [ 1951.812681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.16@tcp -d 192.168.201.16@tcp -m GET -e local_aborted -r 2 [ 1951.865750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.16@tcp -d 192.168.201.16@tcp1 -m GET -e local_aborted -r 2 [ 1951.921846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.16@tcp1 -d 192.168.201.116@tcp -m GET -e local_aborted -r 2 [ 1951.976964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.16@tcp1 -d 192.168.201.116@tcp1 -m GET -e local_aborted -r 2 [ 1952.013923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.16@tcp1 -d 192.168.201.16@tcp -m GET -e local_aborted -r 2 [ 1952.081727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.16@tcp1 -d 192.168.201.16@tcp1 -m GET -e local_aborted -r 2 [ 1952.120934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.16@tcp 192.168.201.116@tcp [ 1952.151302] LNet: There was an unexpected network error while writing to 192.168.201.116: rc = -22 [ 1952.470386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1952.527285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1952.583232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1952.622410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1952.685837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1952.712459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1952.737328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1952.769210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1952.810438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1952.839919] LNet: 88926:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1952.863162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1952.900725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1952.938755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1952.983841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1953.034560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1953.127668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1953.781640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.16@tcp -d 192.168.201.116@tcp -m GET -e local_no_route -r 2 [ 1953.810726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.16@tcp -d 192.168.201.116@tcp1 -m GET -e local_no_route -r 2 [ 1953.851468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.16@tcp -d 192.168.201.16@tcp -m GET -e local_no_route -r 2 [ 1953.900438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.16@tcp -d 192.168.201.16@tcp1 -m GET -e local_no_route -r 2 [ 1953.969881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.16@tcp1 -d 192.168.201.116@tcp -m GET -e local_no_route -r 2 [ 1954.030753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.16@tcp1 -d 192.168.201.116@tcp1 -m GET -e local_no_route -r 2 [ 1954.119825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.16@tcp1 -d 192.168.201.16@tcp -m GET -e local_no_route -r 2 [ 1954.202605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.16@tcp1 -d 192.168.201.16@tcp1 -m GET -e local_no_route -r 2 [ 1954.244410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.16@tcp 192.168.201.116@tcp [ 1954.260435] LNet: There was an unexpected network error while writing to 192.168.201.116: rc = -22 [ 1954.455301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1954.477889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1954.500978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1954.525133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1954.550214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1954.589995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1954.634455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1954.667763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1954.694965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1954.736473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1954.773862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1954.805682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1954.833228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1954.866642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1954.914115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1955.433719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.16@tcp -d 192.168.201.116@tcp -m GET -e local_timeout -r 2 [ 1955.479635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.16@tcp -d 192.168.201.116@tcp1 -m GET -e local_timeout -r 2 [ 1955.527701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.16@tcp -d 192.168.201.16@tcp -m GET -e local_timeout -r 2 [ 1955.597743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.16@tcp -d 192.168.201.16@tcp1 -m GET -e local_timeout -r 2 [ 1955.662824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.16@tcp1 -d 192.168.201.116@tcp -m GET -e local_timeout -r 2 [ 1955.712413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.16@tcp1 -d 192.168.201.116@tcp1 -m GET -e local_timeout -r 2 [ 1955.760557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.16@tcp1 -d 192.168.201.16@tcp -m GET -e local_timeout -r 2 [ 1955.808067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.16@tcp1 -d 192.168.201.16@tcp1 -m GET -e local_timeout -r 2 [ 1955.839760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.16@tcp 192.168.201.116@tcp [ 1956.114647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1956.148616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1956.172611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1956.201784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1956.234521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1956.262700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1956.288814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1956.318587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1956.343133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1956.357625] LNet: 89171:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1956.362474] LNet: 89171:0:(api-ni.c:301:recovery_interval_set()) Skipped 1 previous similar message [ 1956.375890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1956.403252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1956.424275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1956.453369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1956.482876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1956.508381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1956.980274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.16@tcp -d 192.168.201.116@tcp -m GET -e remote_dropped -r 2 [ 1957.006773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.16@tcp -d 192.168.201.116@tcp1 -m GET -e remote_dropped -r 2 [ 1957.035994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.16@tcp -d 192.168.201.16@tcp -m GET -e remote_dropped -r 2 [ 1957.062680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.16@tcp -d 192.168.201.16@tcp1 -m GET -e remote_dropped -r 2 [ 1957.101398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.16@tcp1 -d 192.168.201.116@tcp -m GET -e remote_dropped -r 2 [ 1957.128318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.16@tcp1 -d 192.168.201.116@tcp1 -m GET -e remote_dropped -r 2 [ 1957.167775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.16@tcp1 -d 192.168.201.16@tcp -m GET -e remote_dropped -r 2 [ 1957.203282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.16@tcp1 -d 192.168.201.16@tcp1 -m GET -e remote_dropped -r 2 [ 1957.239737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.16@tcp 192.168.201.116@tcp [ 1957.261495] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.116@tcp [ 1957.527893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1957.557382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1957.598347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1957.641472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1957.688619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1957.741885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1957.778955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1957.827799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1957.866069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1957.894398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1957.937487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1957.994974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1958.028994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1958.064513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1958.111975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1963.997794] LNet: 89367:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1964.024800] LNetError: 89367:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1964.042649] LNet: Removed LNI 192.168.201.16@tcp [ 1964.053986] LNet: Removed LNI 192.168.201.16@tcp1 [ 1964.652766] Key type .llcrypt unregistered [ 1964.654767] Key type ._llcrypt unregistered [ 1983.912492] Key type ._llcrypt registered [ 1983.914628] Key type .llcrypt registered [ 1985.675758] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 18:40:09 (1789598409) [ 1986.385465] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1986.393861] alg: No test for adler32 (adler32-zlib) [ 1987.312179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1987.364726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1987.383102] LNet: Added LNI 192.168.201.16@tcp [8/256/0/180] [ 1988.365825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1988.411167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.201.16@tcp [ 1988.428758] LNetError: 90344:0:(api-ni.c:4478:lnet_ni_set_healthv()) manually adding local NI 192.168.201.16@tcp to recovery [ 1988.518520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.201.16@tcp [ 1988.662755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1995.473367] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 18:40:20 (1789598420) [ 1995.649854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1995.706914] LNet: 90935:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1995.733179] LNetError: 90935:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1995.750462] LNet: Removed LNI 192.168.201.16@tcp [ 1995.792348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1995.857164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1995.887177] LNet: Added LNI 192.168.201.16@tcp [8/256/0/180] [ 1996.709626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1996.725408] LNet: Added LNI 192.168.201.16@tcp1 [8/256/0/180] [ 1997.670828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.201.16@tcp [ 1997.702684] LNetError: 91202:0:(api-ni.c:4478:lnet_ni_set_healthv()) manually adding local NI 192.168.201.16@tcp to recovery [ 1997.856878] LNet: 1 local NIs in recovery (showing 1): 192.168.201.16@tcp [ 1997.859686] LNet: There was an unexpected network error while writing to 192.168.201.16: rc = -22 [ 1998.927201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.201.16@tcp [ 2006.048836] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 18:40:30 (1789598430) [ 2008.318124] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg116-server [ 2010.563442] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 18:40:34 (1789598434) [ 2012.495827] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg116-server [ 2015.005260] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 18:40:39 (1789598439) [ 2017.172608] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg116-server [ 2019.188188] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 18:40:43 (1789598443) [ 2019.524286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2019.578614] LNet: 92825:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2019.595863] LNetError: 92825:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2019.606154] LNet: Removed LNI 192.168.201.16@tcp [ 2019.649170] LNet: Removed LNI 192.168.201.16@tcp1 [ 2019.700484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2019.782490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2019.802891] LNet: Added LNI 192.168.201.16@tcp [8/256/0/180] [ 2020.592739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 2020.675418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 2020.724893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 2020.830653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 2020.881529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 2020.999761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 2027.175618] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 18:40:52 (1789598452) [ 2027.390369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2027.403679] LNet: 93557:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2027.407811] LNetError: 93557:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2027.411595] LNetError: 93557:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2027.431202] LNet: Removed LNI 192.168.201.16@tcp [ 2027.442462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2027.484697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2027.501497] LNet: Added LNI 192.168.201.16@tcp [8/256/0/180] [ 2033.438603] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 18:40:58 (1789598458) [ 2033.561954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2033.583713] LNet: 94257:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2033.587702] LNetError: 94257:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2033.619110] LNet: Removed LNI 192.168.201.16@tcp [ 2033.666964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2033.712874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2033.726816] LNet: Added LNI 192.168.201.16@tcp [8/256/0/180] [ 2040.365160] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 18:41:05 (1789598465) [ 2041.708539] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 2043.154217] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 18:41:08 (1789598468) [ 2043.749509] LNet: 95294:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2043.754351] LNetError: 95294:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2043.762725] LNet: Removed LNI 192.168.201.16@tcp [ 2044.225718] Key type .llcrypt unregistered [ 2044.228266] Key type ._llcrypt unregistered [ 2050.766722] Key type ._llcrypt registered [ 2050.769242] Key type .llcrypt registered [ 2050.931757] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2050.952561] alg: No test for adler32 (adler32-zlib) [ 2051.950663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2052.009396] LNet: Added LNI 192.168.201.16@tcp [8/256/0/180] [ 2059.697276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.116@tcp [ 2065.151506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.201.116@tcp [ 2065.781951] LNet: 95709:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2065.790180] LNetError: 95709:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2065.803328] LNet: Removed LNI 192.168.201.16@tcp [ 2066.323083] Key type .llcrypt unregistered [ 2066.325442] Key type ._llcrypt unregistered [ 2077.710367] Key type ._llcrypt registered [ 2077.712450] Key type .llcrypt registered [ 2078.734650] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 2080.645810] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 2082.312762] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 18:41:47 (1789598507) [ 2083.697767] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 [ 2085.567145] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 18:41:50 (1789598510) [ 2087.330476] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg116-server [ 2089.542334] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 18:41:53 (1789598513) [ 2091.890058] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg116-server [ 2093.565666] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 18:41:58 (1789598518) [ 2095.513064] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2095.572473] alg: No test for adler32 (adler32-zlib) [ 2096.868944] Lustre: Lustre: Build Version: 2.17.58_39_g6aa67ea [ 2097.257414] LNet: Added LNI 192.168.201.16@tcp [8/256/0/180] [ 2099.008793] Key type lgssc registered [ 2100.805450] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2123.691404] Key type lgssc unregistered [ 2123.920927] LNet: 98882:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2123.931579] LNetError: 98882:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2123.951569] LNet: Removed LNI 192.168.201.16@tcp [ 2124.965447] Key type .llcrypt unregistered [ 2124.967678] Key type ._llcrypt unregistered [ 2143.269148] Key type ._llcrypt registered [ 2143.275199] Key type .llcrypt registered [ 2144.563924] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 18:42:49 (1789598569) [ 2145.117610] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2145.145258] alg: No test for adler32 (adler32-zlib) [ 2146.127543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2146.205200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2146.275988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2146.344874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2146.393718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2153.120518] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 18:42:57 (1789598577) [ 2153.381818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2153.399755] LNet: 100294:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2153.418648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2153.460444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2153.481537] LNet: Added LNI 192.168.201.16@tcp [8/256/0/180] [ 2154.161850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2154.174441] LNet: 100439:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2154.177494] LNetError: 100439:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2154.184675] LNet: Removed LNI 192.168.201.16@tcp [ 2154.197715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2154.233580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 2154.251150] LNet: Added LNI 192.168.201.16@tcp [8/256/0/180] [ 2154.722461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2154.749821] LNetError: 100467:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2154.762808] LNet: Removed LNI 192.168.201.16@tcp [ 2154.777283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2154.821952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 2154.846777] LNet: Added LNI 192.168.201.16@tcp [8/256/0/180] [ 2161.200155] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 18:43:06 (1789598586) [ 2161.498460] LNet: 101057:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2161.511921] LNet: 101057:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 2161.524854] LNetError: 101057:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2161.548886] LNet: Removed LNI 192.168.201.16@tcp [ 2162.249314] Key type .llcrypt unregistered [ 2162.252394] Key type ._llcrypt unregistered [ 2168.934797] Key type ._llcrypt registered [ 2168.936991] Key type .llcrypt registered [ 2169.282387] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2169.312276] alg: No test for adler32 (adler32-zlib) [ 2170.379777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2170.512284] LNet: Added LNI 192.168.201.16@tcp [8/256/0/180] [ 2170.515045] LNet: *** cfs_fail_loc=e003, val=100*** [ 2171.792198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.16@tcp [ 2171.823990] LNet: *** cfs_fail_loc=e003, val=100*** [ 2171.828603] LNet: Skipped 2 previous similar messages [ 2171.844890] Ping reply received with mapping: [ 2171.847791] found metadata reply NI[192.168.201.16@tcp] [ 2171.853617] 10 c9 a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 2171.859685] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 2171.866925] 5a 5a 5a 5a 5a 5a 5a 5a [ 2172.039656] LNet: 101452:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2172.059562] LNetError: 101452:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2172.079953] LNet: Removed LNI 192.168.201.16@tcp [ 2173.051689] Key type .llcrypt unregistered [ 2173.056719] Key type ._llcrypt unregistered [ 2189.955602] Key type ._llcrypt registered [ 2189.957771] Key type .llcrypt registered [ 2191.276242] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 18:43:35 (1789598615) [ 2191.706507] Key type .llcrypt unregistered [ 2191.710682] Key type ._llcrypt unregistered [ 2197.085569] Key type ._llcrypt registered [ 2197.087610] Key type .llcrypt registered [ 2197.264257] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2197.273455] alg: No test for adler32 (adler32-zlib) [ 2198.313989] LNetError: 102489:0:(api-ni.c:2855:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 2198.606520] Key type .llcrypt unregistered [ 2198.609526] Key type ._llcrypt unregistered [ 2204.100901] Key type ._llcrypt registered [ 2204.102347] Key type .llcrypt registered [ 2205.573992] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 18:43:50 (1789598630) [ 2206.310303] Key type .llcrypt unregistered [ 2206.313724] Key type ._llcrypt unregistered [ 2213.415869] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2213.425101] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2215.977469] Key type ._llcrypt registered [ 2215.979917] Key type .llcrypt registered [ 2216.285397] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2216.316194] alg: No test for adler32 (adler32-zlib) [ 2217.302522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2217.380061] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2228.371831] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 18:44:12 (1789598652) [ 2228.523274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2228.552909] LNet: 104256:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2228.566437] LNetError: 104256:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2228.594145] LNet: Removed LNI 10.1.2.3@tcp [ 2228.631675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2229.475483] Key type .llcrypt unregistered [ 2229.480644] Key type ._llcrypt unregistered [ 2235.723527] Key type ._llcrypt registered [ 2235.729083] Key type .llcrypt registered [ 2235.822697] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2235.833488] alg: No test for adler32 (adler32-zlib) [ 2236.778253] LNet: Added LNI 192.168.201.16@tcp [8/256/0/180] [ 2236.839782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.16@tcp [ 2236.948971] LNet: 104547:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2236.953457] LNetError: 104547:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2237.993811] LNet: Removed LNI 192.168.201.16@tcp [ 2238.584265] Key type .llcrypt unregistered [ 2238.586447] Key type ._llcrypt unregistered [ 2250.218458] Key type ._llcrypt registered [ 2250.221712] Key type .llcrypt registered [ 2252.015875] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 18:44:36 (1789598676) [ 2252.790451] Key type .llcrypt unregistered [ 2252.794444] Key type ._llcrypt unregistered [ 2258.902286] Key type ._llcrypt registered [ 2258.905359] Key type .llcrypt registered [ 2259.097493] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2259.111880] alg: No test for adler32 (adler32-zlib) [ 2266.935614] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 18:44:51 (1789598691) [ 2267.046571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2267.101740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2267.175255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2267.213925] LNet: Added LNI 192.168.201.16@tcp [8/256/0/180] [ 2268.281738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2268.317783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 2268.382277] LNet: Removed LNI 192.168.201.16@tcp [ 2268.644268] LNet: 106286:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2268.984352] Key type .llcrypt unregistered [ 2268.985822] Key type ._llcrypt unregistered [ 2279.625598] Key type ._llcrypt registered [ 2279.628782] Key type .llcrypt registered [ 2280.875722] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 18:45:05 (1789598705) [ 2281.216478] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2281.234405] alg: No test for adler32 (adler32-zlib) [ 2282.149737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2282.198450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2282.218535] LNet: Added LNI 192.168.201.16@tcp [8/256/0/180] [ 2282.770634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.16@tcp [ 2282.802432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.201.16@tcp [ 2282.816892] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 2282.820266] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 2282.822773] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2282, 0, 0 [ 2282.827487] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 2282.829923] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 2282.831814] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 2282.833457] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2282, 0, 0 [ 2282.836606] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 2282.840041] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2282.843695] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 2282.851123] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2282, 0, 2332 [ 2282.854772] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 2282.856592] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2282.861594] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 2282.865745] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2282, 0, 2332 [ 2282.870612] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 2288.914464] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 18:45:13 (1789598713) [ 2289.556494] LNet: 107824:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2289.560847] LNetError: 107824:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2290.605927] LNet: Removed LNI 192.168.201.16@tcp [ 2291.165604] Key type .llcrypt unregistered [ 2291.168259] Key type ._llcrypt unregistered [ 2298.648244] Key type ._llcrypt registered [ 2298.650804] Key type .llcrypt registered [ 2298.842326] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2298.853542] alg: No test for adler32 (adler32-zlib) [ 2299.793993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2299.832827] LNet: Added LNI 192.168.201.16@tcp [8/256/0/180] [ 2308.796500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.116@tcp [ 2318.641222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2318.730932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2318.781177] LNet: Added LNI 192.168.201.16@tcp1 [8/256/0/180] [ 2320.125269] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2320.373360] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2320.376354] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2320.382632] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2322.897274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 2322.915856] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 2335.465001] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 18:45:59 (1789598759) [ 2335.727176] LNet: 109323:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2335.744025] LNetError: 109323:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2336.800746] LNet: Removed LNI 192.168.201.16@tcp [ 2336.815338] LNetError: 109323:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2336.825971] LNet: Removed LNI 192.168.201.16@tcp1 [ 2336.845630] LNet: Removed LNI 10.1.2.3@tcp99 [ 2337.804747] Key type .llcrypt unregistered [ 2337.808655] Key type ._llcrypt unregistered [ 2345.573797] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2345.578487] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2348.320652] Key type ._llcrypt registered [ 2348.326844] Key type .llcrypt registered [ 2348.589657] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2348.633075] alg: No test for adler32 (adler32-zlib) [ 2349.648488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2349.711518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2349.732454] LNet: Added LNI 192.168.201.16@tcp [8/256/0/180] [ 2350.408918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2350.421589] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2352.336594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 2352.397144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2352.453943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 2352.471578] LNetError: 110067:0:(peer.c:2366:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 2352.489351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2352.508376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.16@tcp [ 2352.756268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2352.824388] LNet: 110087:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2352.832508] LNetError: 110087:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2352.840278] LNet: Removed LNI 10.1.2.3@tcp [ 2353.897624] LNet: Removed LNI 192.168.201.16@tcp [ 2354.742632] Key type .llcrypt unregistered [ 2354.745506] Key type ._llcrypt unregistered [ 2366.689125] Key type ._llcrypt registered [ 2366.691226] Key type .llcrypt registered [ 2367.770978] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 18:46:32 (1789598792) [ 2368.264563] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2368.274549] alg: No test for adler32 (adler32-zlib) [ 2369.137260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2369.176221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2369.191629] LNet: Added LNI 192.168.201.16@tcp [8/256/0/180] [ 2369.749838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.16@tcp [ 2369.802895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg116-client@tcp [ 2376.258616] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 18:46:40 (1789598800) [ 2376.358176] LNet: 111613:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2376.370355] LNetError: 111613:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2377.446420] LNet: Removed LNI 192.168.201.16@tcp [ 2378.164585] Key type .llcrypt unregistered [ 2378.166449] Key type ._llcrypt unregistered [ 2386.742660] Key type ._llcrypt registered [ 2386.748095] Key type .llcrypt registered [ 2387.186935] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2387.265266] alg: No test for adler32 (adler32-zlib) [ 2388.315177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 2388.350980] LNet: Added LNI 192.168.201.16@tcp [8/256/0/180] [ 2388.412211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 2398.296195] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 18:47:02 (1789598822) [ 2399.722980] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 2402.011607] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 18:47:06 (1789598826) [ 2403.825681] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 2405.754330] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 18:47:10 (1789598830) [ 2405.882336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2405.893843] LNet: 113053:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2405.897845] LNetError: 113053:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2405.919232] LNet: Removed LNI 192.168.201.16@tcp [ 2405.928677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2405.959882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2405.988242] LNet: Added LNI 192.168.201.16@tcp [8/256/0/180] [ 2406.945585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2407.017655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2415.311403] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 18:47:19 (1789598839) [ 2415.485786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2415.526146] LNet: 113761:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2415.542759] LNetError: 113761:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2415.578141] LNet: Removed LNI 192.168.201.16@tcp [ 2415.609879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2415.707680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 2415.760421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2416.634379] Key type .llcrypt unregistered [ 2416.639613] Key type ._llcrypt unregistered [ 2430.362456] Key type ._llcrypt registered [ 2430.364142] Key type .llcrypt registered [ 2431.811426] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 18:47:36 (1789598856) [ 2432.314064] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2432.336272] alg: No test for adler32 (adler32-zlib) [ 2433.431046] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2433.519595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2433.547457] LNet: Added LNI 192.168.201.16@tcp [8/256/0/180] [ 2434.504772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2434.546387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.16@tcp [ 2442.126129] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 18:47:46 (1789598866) [ 2442.217841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2442.231192] LNet: 115313:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2442.239994] LNetError: 115313:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2442.266865] LNet: Removed LNI 192.168.201.16@tcp [ 2442.307829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2442.359815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 2442.387885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 2450.103379] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 18:47:54 (1789598874) [ 2450.242849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2450.282961] LNet: 115891:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2450.318717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2450.482352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2451.102934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2451.133246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2451.197261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 2451.566466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2451.604129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2451.706542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2451.759545] LNet: 115932:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2451.806721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2451.906725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2452.507415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2452.567655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2452.631566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 2453.185670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2453.219463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2460.380901] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 18:48:05 (1789598885) [ 2460.638914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2460.674367] LNet: 116531:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2460.709097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2460.819349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2460.854765] LNet: Added LNI 192.168.201.16@tcp [8/256/0/180] [ 2461.527847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2461.551423] LNet: Added LNI 192.168.201.16@tcp2 [8/256/0/180] [ 2462.279242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2462.327204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.16@tcp [ 2462.524935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2462.538973] LNetError: 116812:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2463.585771] LNet: Removed LNI 192.168.201.16@tcp [ 2463.590537] LNetError: 116812:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2463.605058] LNet: Removed LNI 192.168.201.16@tcp2 [ 2463.648909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2463.728310] LNet: Added LNI 192.168.201.16@tcp [8/256/0/180] [ 2463.745885] LNet: Added LNI 192.168.201.16@tcp2 [8/256/0/180] [ 2463.758538] LNet: 116818:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2464.452481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.16@tcp [ 2464.769878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2464.796435] LNet: 116855:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2464.806874] LNet: 116855:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 2465.824265] LNet: Removed LNI 192.168.201.16@tcp [ 2465.825855] LNetError: 116855:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2465.833736] LNetError: 116855:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2465.850340] LNet: Removed LNI 192.168.201.16@tcp2 [ 2465.892542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2465.954314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2465.978522] LNet: Added LNI 192.168.201.16@tcp [8/256/0/180] [ 2466.741482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2466.756764] LNet: Added LNI 192.168.201.16@tcp2 [8/256/0/180] [ 2467.477295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.16@tcp [ 2467.509430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2467.563284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.16@tcp [ 2467.745703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2468.769091] LNet: Removed LNI 192.168.201.16@tcp [ 2469.799336] LNet: Removed LNI 192.168.201.16@tcp2 [ 2469.824668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2469.933799] LNet: Added LNI 192.168.201.16@tcp [8/256/0/180] [ 2469.954796] LNet: Added LNI 192.168.201.16@tcp2 [8/256/0/180] [ 2469.966316] LNet: 117146:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2470.655765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.16@tcp [ 2470.890163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2470.934108] LNetError: 117183:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2470.940618] LNetError: 117183:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 2470.974466] LNet: Removed LNI 192.168.201.16@tcp [ 2472.041347] LNet: Removed LNI 192.168.201.16@tcp2 [ 2472.129612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2472.235544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2472.277593] LNet: Added LNI 192.168.201.16@tcp [8/256/0/180] [ 2473.308644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2473.345519] LNet: Added LNI 192.168.201.16@tcp2 [8/256/0/180] [ 2474.369512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2474.439737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.16@tcp [ 2474.722604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2474.744760] LNet: 117463:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2474.750911] LNet: 117463:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 2475.808231] LNet: Removed LNI 192.168.201.16@tcp [ 2475.819649] LNet: Removed LNI 192.168.201.16@tcp2 [ 2475.866590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2475.943934] LNet: Added LNI 192.168.201.16@tcp [8/256/0/180] [ 2475.950553] LNet: Added LNI 192.168.201.16@tcp2 [8/256/0/180] [ 2475.957369] LNet: 117469:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2476.574798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.16@tcp [ 2476.833389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2477.920532] LNet: Removed LNI 192.168.201.16@tcp [ 2477.932801] LNet: Removed LNI 192.168.201.16@tcp2 [ 2477.957739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2478.005494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2478.054379] LNet: Added LNI 192.168.201.16@tcp [8/256/0/180] [ 2478.809225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2478.827318] LNet: Added LNI 192.168.201.16@tcp2 [8/256/0/180] [ 2479.764602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.16@tcp [ 2479.796384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2479.841679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.16@tcp [ 2480.000232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2480.019096] LNetError: 117790:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2480.025149] LNetError: 117790:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 2480.030732] LNet: Removed LNI 192.168.201.16@tcp [ 2481.074982] LNet: Removed LNI 192.168.201.16@tcp2 [ 2481.150235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2481.236809] LNet: Added LNI 192.168.201.16@tcp [8/256/0/180] [ 2481.244292] LNet: Added LNI 192.168.201.16@tcp2 [8/256/0/180] [ 2481.251440] LNet: 117797:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2481.908813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.16@tcp [ 2482.154682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2482.174417] LNet: Removed LNI 192.168.201.16@tcp [ 2483.245473] LNet: Removed LNI 192.168.201.16@tcp2 [ 2483.307848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2483.391967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2483.420239] LNet: Added LNI 192.168.201.16@tcp [8/256/0/180] [ 2484.452619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2484.496396] LNet: Added LNI 192.168.201.16@tcp2 [8/256/0/180] [ 2485.319913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2485.347937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2485.413262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.16@tcp [ 2485.830282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2486.880210] LNet: Removed LNI 192.168.201.16@tcp [ 2486.892316] LNet: Removed LNI 192.168.201.16@tcp2 [ 2486.943486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2487.026104] LNet: Added LNI 192.168.201.16@tcp [8/256/0/180] [ 2487.038283] LNet: Added LNI 192.168.201.16@tcp2 [8/256/0/180] [ 2487.046930] LNet: 118136:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2487.773389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.16@tcp [ 2488.190679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2489.248254] LNet: Removed LNI 192.168.201.16@tcp [ 2489.263370] LNet: Removed LNI 192.168.201.16@tcp2 [ 2489.324804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2489.430436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2489.459531] LNet: Added LNI 192.168.201.16@tcp [8/256/0/180] [ 2490.236589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2490.255360] LNet: Added LNI 192.168.201.16@tcp2 [8/256/0/180] [ 2490.835466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.16@tcp [ 2490.869187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2490.903300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2490.965565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.16@tcp [ 2491.359743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2491.384824] LNet: 118490:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2491.392451] LNet: 118490:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 5 previous similar messages [ 2492.448286] LNet: Removed LNI 192.168.201.16@tcp [ 2493.482661] LNet: Removed LNI 192.168.201.16@tcp2 [ 2493.508902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2493.551463] LNet: Added LNI 192.168.201.16@tcp [8/256/0/180] [ 2493.555300] LNet: Added LNI 192.168.201.16@tcp2 [8/256/0/180] [ 2493.908452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.16@tcp [ 2501.326438] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 2502.716399] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 18:48:47 (1789598927) [ 2502.889917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2502.909546] LNetError: 119256:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2502.912258] LNetError: 119256:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 9 previous similar messages [ 2502.920605] LNet: Removed LNI 192.168.201.16@tcp [ 2503.979673] LNet: Removed LNI 192.168.201.16@tcp2 [ 2504.028651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2504.485199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2504.528990] LNet: Added LNI 192.168.201.16@tcp [8/256/0/180] [ 2505.117954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2505.135898] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2505.796878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2505.814064] LNet: Removed LNI 10.1.2.3@tcp [ 2505.819821] LNet: Removed LNI 192.168.201.16@tcp [ 2505.828885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2505.867984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 2505.896652] LNet: Added LNI 192.168.201.16@tcp [8/256/0/180] [ 2505.901589] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2506.229312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 2506.248233] LNet: Removed LNI 192.168.201.16@tcp [ 2506.270914] LNet: Removed LNI 10.1.2.3@tcp [ 2513.183545] Lustre: DEBUG MARKER: == sanity-lnet test 475: per-operation latency stats populate and reset ========================================================== 18:48:57 (1789598937) [ 2513.539984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2513.602721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2513.647317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2513.663709] LNet: Added LNI 192.168.201.16@tcp [8/256/0/180] [ 2514.306552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2514.353081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2514.395308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.16@tcp [ 2514.441789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.16@tcp [ 2514.510130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.16@tcp [ 2514.571853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.16@tcp [ 2514.645198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.16@tcp [ 2514.686442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.16@tcp [ 2514.748808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.16@tcp [ 2514.807311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.16@tcp [ 2514.846469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.16@tcp [ 2514.891077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.16@tcp [ 2515.088181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2522.113341] Lustre: DEBUG MARKER: == sanity-lnet test 476: lnetctl set latency_stats toggles capture ========================================================== 18:49:06 (1789598946) [ 2522.444629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2523.501300] LNet: Removed LNI 192.168.201.16@tcp [ 2523.531253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2523.590465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2523.618785] LNet: Added LNI 192.168.201.16@tcp [8/256/0/180] [ 2524.574549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 0 [ 2524.629549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2524.673569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.16@tcp [ 2524.704520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.16@tcp [ 2524.742424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.16@tcp [ 2524.790380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.16@tcp [ 2524.860296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.16@tcp [ 2524.935580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.16@tcp [ 2524.988177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.16@tcp [ 2525.033410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.16@tcp [ 2525.109675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.16@tcp [ 2525.182414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.16@tcp [ 2525.403603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2525.505427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.16@tcp [ 2525.593696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.16@tcp [ 2525.670217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.16@tcp [ 2525.707924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.16@tcp [ 2525.753827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.16@tcp [ 2525.816064] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.16@tcp [ 2525.850475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.16@tcp [ 2525.886371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.16@tcp [ 2525.936124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.16@tcp [ 2525.968631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.16@tcp [ 2533.574333] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 18:49:17 (1789598957) [ 2533.698596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2533.716658] LNet: 121620:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2533.721533] LNet: 121620:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 2534.765109] LNet: Removed LNI 192.168.201.16@tcp [ 2534.842634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2535.041899] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2535.184415] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2535.187232] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2535.189825] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2537.564254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2537.593213] LNet: Added LNI 192.168.201.16@tcp [8/256/0/180] [ 2538.462366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2538.480761] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2541.146821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.16@tcp [ 2541.234625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 2541.402476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.201.16@tcp -m PUT -e local_timeout -r 1 [ 2541.508878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 2541.670218] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2541.681469] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2542.061650] LNetError: 121685:0:(lib-move.c:4794:lnet_parse()) 192.168.201.16@tcp, src 192.168.201.16@tcp: Dropping PUT (error -108 looking up sender) [ 2543.073201] LNetError: 122146:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2543.075810] LNetError: 122146:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 2543.093389] LNet: Removed LNI 10.1.2.3@tcp [ 2544.166655] LNet: Removed LNI 192.168.201.16@tcp [ 2544.898405] Key type .llcrypt unregistered [ 2544.900996] Key type ._llcrypt unregistered [ 2561.997695] Key type ._llcrypt registered [ 2561.999154] Key type .llcrypt registered [ 2563.111803] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 18:49:48 (1789598988) [ 2563.608231] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2563.622226] alg: No test for adler32 (adler32-zlib) [ 2564.610934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2565.123569] LNet: 122996:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2565.974433] Key type .llcrypt unregistered [ 2565.976921] Key type ._llcrypt unregistered [ 2573.484358] Key type ._llcrypt registered [ 2573.486214] Key type .llcrypt registered [ 2573.726996] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2573.743190] alg: No test for adler32 (adler32-zlib) [ 2574.703865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2574.732448] LNet: Added LNI 192.168.201.16@tcp [8/256/0/180] [ 2583.657123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.116@tcp [ 2583.895280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 2584.747586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.201.116@tcp [ 2584.961383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2584.986268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2584.993373] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.116@tcp [ 2585.013966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2585.041779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2585.068410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2585.101496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2585.145476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2585.171901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2585.198522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2585.213851] LNet: 123444:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2585.232896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2585.260820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2585.293575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2585.335464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2585.366134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2585.392848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2586.140977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2586.356335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2586.377161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2586.401465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2586.423871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2586.444719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2586.469464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2586.491223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2586.515844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2586.538347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2586.550328] LNet: 123563:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2586.571367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2586.609268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2586.642313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2586.676516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2586.703431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2586.730691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2587.355756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.201.116@tcp [ 2587.541752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2587.567469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2587.591139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2587.623362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2587.674682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2587.723375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2587.754682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2587.782932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2587.824388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2587.839412] LNet: 123682:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2587.861338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2587.900983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2587.949574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2587.987833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2588.037826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2588.075201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2588.725603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2588.940153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2588.957892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2588.976389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2588.998693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2589.029968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2589.049939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2589.086251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2589.115421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2589.159662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2589.192235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2589.216501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2589.241406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2589.268986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2589.305257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2589.340741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2589.743426] LNet: 123848:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2589.747605] LNetError: 123848:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2589.763204] LNet: Removed LNI 192.168.201.16@tcp [ 2590.230049] Key type .llcrypt unregistered [ 2590.233948] Key type ._llcrypt unregistered [ 2609.399965] Key type ._llcrypt registered [ 2609.402174] Key type .llcrypt registered [ 2610.695795] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 18:50:35 (1789599035) [ 2611.085913] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2611.110596] alg: No test for adler32 (adler32-zlib) [ 2612.146975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2612.557197] LNet: 124733:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2613.048523] Key type .llcrypt unregistered [ 2613.051598] Key type ._llcrypt unregistered [ 2620.030279] Key type ._llcrypt registered [ 2620.032494] Key type .llcrypt registered [ 2620.179387] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2620.205427] alg: No test for adler32 (adler32-zlib) [ 2621.191446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2621.235792] LNet: Added LNI 192.168.201.16@tcp [8/256/0/180] [ 2628.842381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.116@tcp [ 2638.998515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2639.067424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2639.098571] LNet: Added LNI 192.168.201.16@tcp1 [8/256/0/180] [ 2640.111297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 2640.615391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.201.116@tcp [ 2640.747372] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.116@tcp [ 2640.894747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2640.932924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2640.970147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2641.022497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2641.062386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2641.098630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2641.123874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2641.187423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2641.232572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2641.250510] LNet: 125384:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2641.271720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2641.299489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2641.338601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2641.384373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2641.415547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2641.445158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2642.190070] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2642.416260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2642.437055] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2642.461262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2642.482756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2642.525616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2642.572995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2642.597713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2642.624542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2642.658700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2642.677310] LNet: 125503:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2642.695060] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2642.740027] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2642.800459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2642.837838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2642.873501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2642.915424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2643.570036] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.201.116@tcp [ 2643.793571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2643.821768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2643.847813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2643.875818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2643.907665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2643.950770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2644.001270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2644.040253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2644.079731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2644.105767] LNet: 125622:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2644.135844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2644.187056] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2644.254552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2644.290378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2644.315995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2644.362948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2645.058537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2645.319996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2645.349961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2645.376561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2645.406981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2645.465819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2645.523333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2645.578279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2645.630283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2645.717794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2645.828334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2645.880395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2645.916851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2645.953493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2645.983681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2646.038605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2651.946409] LNet: 125823:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2651.956864] LNetError: 125823:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2651.968224] LNet: Removed LNI 192.168.201.16@tcp [ 2651.981280] LNet: Removed LNI 192.168.201.16@tcp1 [ 2652.531985] Key type .llcrypt unregistered [ 2652.534592] Key type ._llcrypt unregistered [ 2674.736448] Key type ._llcrypt registered [ 2674.737804] Key type .llcrypt registered [ 2676.057934] Lustre: DEBUG MARKER: == sanity-lnet test 525: keep reachable peer on local net delete ========================================================== 18:51:40 (1789599100) [ 2676.776772] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2676.793733] alg: No test for adler32 (adler32-zlib) [ 2677.681548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2677.725250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2677.747719] LNet: Added LNI 192.168.201.16@tcp [8/256/0/180] [ 2678.500101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2678.517987] LNet: Added LNI 192.168.201.16@tcp2 [8/256/0/180] [ 2679.381471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.16@tcp [ 2679.456642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp [ 2679.479177] LNet: Removed LNI 192.168.201.16@tcp [ 2686.711119] Lustre: DEBUG MARKER: == sanity-lnet test 550: DD-off: keep route up when gw reachable via another local NI ========================================================== 18:51:51 (1789599111) [ 2688.237148] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_550 Need at least 2 local interfaces for multi-rail [ 2690.070950] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 18:51:55 (1789599115) [ 2699.117479] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 18:52:03 (1789599123) [ 2707.279674] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 18:52:12 (1789599132) [ 2715.571442] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 18:52:20 (1789599140) [ 2723.275776] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 18:52:28 (1789599148) [ 2731.377879] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 18:52:35 (1789599155) [ 2739.568772] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 18:52:44 (1789599164) [ 2747.563492] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 18:52:52 (1789599172) [ 2756.295184] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 18:53:00 (1789599180) [ 2764.670345] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 18:53:09 (1789599189) [ 2771.921971] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 18:53:16 (1789599196) [ 2779.952276] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 18:53:24 (1789599204) [ 2787.281812] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 18:53:32 (1789599212) [ 2795.502446] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 18:53:39 (1789599219) [ 2805.509412] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 18:53:49 (1789599229) [ 2816.139696] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 18:54:00 (1789599240) [ 2824.542723] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 18:54:09 (1789599249) [ 2832.635850] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 18:54:17 (1789599257) [ 2832.872285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2832.895450] LNet: 137517:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2832.902495] LNetError: 137517:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2832.917355] LNet: Removed LNI 192.168.201.16@tcp2 [ 2832.932937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2833.087722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2833.133147] LNet: Added LNI 192.168.201.16@tcp [8/256/0/180] [ 2834.476220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2834.509503] LNet: 137659:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2834.515834] LNetError: 137659:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2834.529255] LNet: Removed LNI 192.168.201.16@tcp [ 2834.543448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2834.618779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2834.641725] LNet: Added LNI 192.168.201.16@tcp [8/256/0/180] [ 2842.694291] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 18:54:26 (1789599266) [ 2850.270433] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 18:54:35 (1789599275) [ 2858.080147] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 18:54:42 (1789599282) [ 2865.920648] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 18:54:50 (1789599290) [ 2873.548166] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 18:54:58 (1789599298) [ 2881.410737] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 18:55:06 (1789599306) [ 2889.016095] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 18:55:13 (1789599313) [ 2896.714163] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 18:55:21 (1789599321) [ 2904.440945] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 18:55:28 (1789599328) [ 2911.982671] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 18:55:37 (1789599337) [ 2919.369702] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 18:55:44 (1789599344) [ 2926.677405] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 18:55:51 (1789599351) [ 2934.631197] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 2646 sec ========== 18:55:59 (1789599359) [ 2934.879931] LNet: 145125:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2934.894990] LNetError: 145125:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2934.911600] LNet: Removed LNI 192.168.201.16@tcp [ 2935.651740] Key type .llcrypt unregistered [ 2935.655939] Key type ._llcrypt unregistered [ 2948.654276] Key type ._llcrypt registered [ 2948.656041] Key type .llcrypt registered [ 2949.464370] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2949.496105] alg: No test for adler32 (adler32-zlib) [ 2950.778743] Lustre: Lustre: Build Version: 2.17.58_39_g6aa67ea [ 2951.167718] LNet: Added LNI 192.168.201.16@tcp [8/256/0/180] [ 2952.904285] Key type lgssc registered [ 2954.239853] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3023.076277] Lustre: Mounted lustre-client - version 2.17.58_39_g6aa67ea [ 3030.427726] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3039.569542] Lustre: Unmounted lustre-client [ 3058.445344] Key type lgssc unregistered [ 3058.677967] LNet: 148074:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3058.683587] LNetError: 148074:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3058.700307] LNet: Removed LNI 192.168.201.16@tcp [ 3059.617156] Key type .llcrypt unregistered [ 3059.622716] Key type ._llcrypt unregistered