[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 441511875 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.002318] x2apic enabled [ 0.003007] Switched APIC routing to physical x2apic. [ 0.004012] kvm-guest: setup PV IPIs [ 0.007000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007024] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008016] pid_max: default: 32768 minimum: 301 [ 0.009152] LSM: Security Framework initializing [ 0.010063] Yama: becoming mindful. [ 0.011035] SELinux: Initializing. [ 0.012076] *** VALIDATE selinux *** [ 0.020550] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.025388] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027154] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028117] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029157] *** VALIDATE tmpfs *** [ 0.031083] *** VALIDATE proc *** [ 0.032307] *** VALIDATE cgroup *** [ 0.033007] *** VALIDATE cgroup2 *** [ 0.034324] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.035178] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.036018] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.037030] Spectre V2 : User space: Vulnerable [ 0.038012] Speculative Store Bypass: Vulnerable [ 0.041313] debug: unmapping init [mem 0xffffffffa1659000-0xffffffffa1660fff] [ 0.043208] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.044727] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.045025] ... version: 2 [ 0.046015] ... bit width: 48 [ 0.047015] ... generic registers: 4 [ 0.048016] ... value mask: 0000ffffffffffff [ 0.049019] ... max period: 00007fffffffffff [ 0.050017] ... fixed-purpose events: 3 [ 0.051013] ... event mask: 000000070000000f [ 0.052309] rcu: Hierarchical SRCU implementation. [ 0.054580] smp: Bringing up secondary CPUs ... [ 0.055737] x86: Booting SMP configuration: [ 0.056033] .... node #0, CPUs: #1 #2 #3 [ 0.059247] smp: Brought up 1 node, 4 CPUs [ 0.061010] smpboot: Max logical packages: 1 [ 0.062018] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.153140] node 0 deferred pages initialised in 89ms [ 0.156688] devtmpfs: initialized [ 0.158425] x86/mm: Memory block size: 128MB [ 0.162040] gcov: version magic: 0x41383552 [ 0.164279] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.165063] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.166337] pinctrl core: initialized pinctrl subsystem [ 0.167207] [ 0.167835] ************************************************************* [ 0.168015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.169015] ** ** [ 0.170027] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.171015] ** ** [ 0.172013] ** This means that this kernel is built to expose internal ** [ 0.173016] ** IOMMU data structures, which may compromise security on ** [ 0.174015] ** your system. ** [ 0.175015] ** ** [ 0.176014] ** If you see this message and you are not debugging the ** [ 0.177018] ** kernel, report this immediately to your vendor! ** [ 0.178015] ** ** [ 0.179015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.180017] ************************************************************* [ 0.181762] NET: Registered protocol family 16 [ 0.182536] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.183149] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.184066] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.186030] cpuidle: using governor menu [ 0.188800] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.192487] PCI: Using configuration type 1 for base access [ 0.194185] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.204220] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.208033] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.212078] cryptd: max_cpu_qlen set to 1000 [ 0.216218] ACPI: Added _OSI(Module Device) [ 0.218019] ACPI: Added _OSI(Processor Device) [ 0.220070] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.222018] ACPI: Added _OSI(Processor Aggregator Device) [ 0.227964] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.235672] ACPI: Interpreter enabled [ 0.237299] ACPI: PM: (supports S0 S3 S4 S5) [ 0.239020] ACPI: Using IOAPIC for interrupt routing [ 0.242179] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.246736] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.260615] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.263067] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.267043] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.272420] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.278686] acpiphp: Slot [2] registered [ 0.280150] acpiphp: Slot [5] registered [ 0.282161] acpiphp: Slot [6] registered [ 0.284144] acpiphp: Slot [3] registered [ 0.285000] acpiphp: Slot [4] registered [ 0.285000] acpiphp: Slot [7] registered [ 0.288136] acpiphp: Slot [8] registered [ 0.290110] acpiphp: Slot [9] registered [ 0.291215] acpiphp: Slot [10] registered [ 0.293104] acpiphp: Slot [11] registered [ 0.295103] acpiphp: Slot [12] registered [ 0.296103] acpiphp: Slot [13] registered [ 0.298114] acpiphp: Slot [14] registered [ 0.299158] acpiphp: Slot [15] registered [ 0.301120] acpiphp: Slot [16] registered [ 0.302073] acpiphp: Slot [17] registered [ 0.303087] acpiphp: Slot [18] registered [ 0.305091] acpiphp: Slot [19] registered [ 0.306102] acpiphp: Slot [20] registered [ 0.307239] acpiphp: Slot [21] registered [ 0.309109] acpiphp: Slot [22] registered [ 0.311124] acpiphp: Slot [23] registered [ 0.313177] acpiphp: Slot [24] registered [ 0.314104] acpiphp: Slot [25] registered [ 0.316105] acpiphp: Slot [26] registered [ 0.318114] acpiphp: Slot [27] registered [ 0.319100] acpiphp: Slot [28] registered [ 0.321370] acpiphp: Slot [29] registered [ 0.324151] acpiphp: Slot [30] registered [ 0.326104] acpiphp: Slot [31] registered [ 0.328071] PCI host bridge to bus 0000:00 [ 0.329019] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.332029] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.335026] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.338030] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.341028] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.344038] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.347190] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.350072] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.353830] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.363015] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.368417] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.369024] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.372020] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.374017] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.377437] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.379594] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.382031] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.383755] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.387000] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.396015] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.401016] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.405778] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.415019] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.427017] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.448018] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.457904] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.464021] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.469016] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.478015] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.485591] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.486269] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.489719] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.491308] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.492197] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.496169] iommu: Default domain type: Passthrough [ 0.497321] SCSI subsystem initialized [ 0.498090] ACPI: bus type USB registered [ 0.499083] usbcore: registered new interface driver usbfs [ 0.501087] usbcore: registered new interface driver hub [ 0.504084] usbcore: registered new device driver usb [ 0.505230] pps_core: LinuxPPS API ver. 1 registered [ 0.506007] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.509048] PTP clock support registered [ 0.510127] EDAC MC: Ver: 3.0.0 [ 0.513026] PCI: Using ACPI for IRQ routing [ 0.514508] NetLabel: Initializing [ 0.516014] NetLabel: domain hash size = 128 [ 0.517011] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.519083] NetLabel: unlabeled traffic allowed by default [ 0.522068] vgaarb: loaded [ 0.523286] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.524014] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.531000] clocksource: Switched to clocksource kvm-clock [ 0.622194] VFS: Disk quotas dquot_6.6.0 [ 0.626344] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.628817] *** VALIDATE ramfs *** [ 0.633276] *** VALIDATE hugetlbfs *** [ 0.636543] pnp: PnP ACPI init [ 0.641318] pnp: PnP ACPI: found 6 devices [ 0.681556] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.690437] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.695409] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.700674] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.711177] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.716600] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.720821] NET: Registered protocol family 2 [ 0.723915] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.737855] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.749269] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.768984] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.779427] TCP: Hash tables configured (established 65536 bind 65536) [ 0.794530] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.807931] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.812843] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.818408] NET: Registered protocol family 1 [ 0.822537] RPC: Registered named UNIX socket transport module. [ 0.825825] RPC: Registered udp transport module. [ 0.832907] RPC: Registered tcp transport module. [ 0.834855] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.837898] NET: Registered protocol family 44 [ 0.840521] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.843419] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.846267] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.849330] PCI: CLS 0 bytes, default 64 [ 0.850914] Unpacking initramfs... [ 4.526786] debug: unmapping init [mem 0xffff8ead7cc64000-0xffff8ead7ffcffff] [ 4.534722] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 4.537204] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 4.541590] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 6.046362] Initialise system trusted keyrings [ 6.053045] Key type blacklist registered [ 6.061093] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 6.086945] zbud: loaded [ 6.092357] *** VALIDATE nfs *** [ 6.096228] *** VALIDATE nfs4 *** [ 6.099318] pstore: using deflate compression [ 6.132197] Platform Keyring initialized [ 6.470579] NET: Registered protocol family 38 [ 6.472363] Key type asymmetric registered [ 6.473854] Asymmetric key parser 'x509' registered [ 6.475982] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 6.478658] io scheduler mq-deadline registered [ 6.480583] io scheduler kyber registered [ 6.482244] io scheduler bfq registered [ 6.483852] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 6.487139] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 6.489904] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 6.502165] ACPI: Power Button [PWRF] [ 6.518097] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 6.536658] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 6.563671] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 6.601417] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 6.641205] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 6.650342] Non-volatile memory driver v1.3 [ 6.654270] Linux agpgart interface v0.103 [ 6.702908] virtio_blk virtio1: [vda] 145904 512-byte logical blocks (74.7 MB/71.2 MiB) [ 6.713280] vda: detected capacity change from 0 to 74702848 [ 6.739792] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 6.746065] vdb: detected capacity change from 0 to 1073741824 [ 6.755409] libphy: Fixed MDIO Bus: probed [ 6.764967] usbcore: registered new interface driver usbserial_generic [ 6.768350] usbserial: USB Serial support registered for generic [ 6.771242] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 6.777217] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 6.779402] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 6.782217] mousedev: PS/2 mouse device common for all mice [ 6.785797] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 6.789071] rtc_cmos 00:05: RTC can wake from S4 [ 6.793990] rtc_cmos 00:05: registered as rtc0 [ 6.797734] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 6.799654] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 6.806106] intel_pstate: CPU model not supported [ 6.815336] hid: raw HID events driver (C) Jiri Kosina [ 6.817163] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 6.820659] usbcore: registered new interface driver usbhid [ 6.820665] usbhid: USB HID core driver [ 6.820794] drop_monitor: Initializing network drop monitor service [ 6.820943] Initializing XFRM netlink socket [ 6.821320] NET: Registered protocol family 10 [ 6.822492] Segment Routing with IPv6 [ 6.822525] NET: Registered protocol family 17 [ 6.822718] mpls_gso: MPLS GSO support [ 6.848246] RAS: Correctable Errors collector initialized. [ 6.850825] AVX version of gcm_enc/dec engaged. [ 6.852689] AES CTR mode by8 optimization enabled [ 7.013604] sched_clock: Marking stable (7013586496, 0)->(7993844061, -980257565) [ 7.033799] registered taskstats version 1 [ 7.036421] Loading compiled-in X.509 certificates [ 7.039276] zswap: loaded using pool lzo/zbud [ 7.105370] Key type big_key registered [ 7.147375] Key type encrypted registered [ 7.152205] ima: No TPM chip found, activating TPM-bypass! [ 7.156330] ima: Allocated hash algorithm: sha1 [ 7.158214] ima: No architecture policies found [ 7.171707] evm: Initialising EVM extended attributes: [ 7.174980] evm: security.selinux [ 7.177579] evm: security.ima [ 7.179285] evm: security.capability [ 7.180648] evm: HMAC attrs: 0x1 [ 7.186909] rtc_cmos 00:05: setting system clock to 2026-08-19 05:35:10 UTC (1787117710) [ 7.206614] debug: unmapping init [mem 0xffffffffa2603000-0xffffffffa27fffff] [ 7.216827] debug: unmapping init [mem 0xffffffffa1382000-0xffffffffa1658fff] [ 7.242188] Write protecting the kernel read-only data: 28672k [ 7.263134] debug: unmapping init [mem 0xffffffff9fa03000-0xffffffff9fbfffff] [ 7.270315] debug: unmapping init [mem 0xffffffffa0314000-0xffffffffa03fffff] [ 7.314413] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 7.329133] systemd[1]: Detected virtualization kvm. [ 7.332054] systemd[1]: Detected architecture x86-64. [ 7.334763] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 7.364722] systemd[1]: No hostname configured. [ 7.366796] systemd[1]: Set hostname to . [ 7.369403] random: systemd: uninitialized urandom read (16 bytes read) [ 7.375146] systemd[1]: Initializing machine ID from random generator. [ 7.509474] random: ln: uninitialized urandom read (6 bytes read) [ 7.657366] random: systemd: uninitialized urandom read (16 bytes read) [ 7.659809] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 7.667575] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 7.681386] systemd[1]: Started Memstrack Anylazing Service. [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Timers. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on udev Control Socket. Starting Apply Kernel Variables... [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Sockets. Starting Setup Virtual Console... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. Starting Create list of required st…ce nodes for the current kernel... Starting Journal Service... [ OK ] Reached target Swap. [ OK ] Reached target Slices. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 9.173025] device-mapper: uevent: version 1.0.3 [ 9.180649] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 11.021164] virtio_net virtio0 ens2: renamed from eth0 [ 11.290447] scsi host0: ata_piix [ 11.312331] scsi host1: ata_piix [ 11.316114] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 11.322487] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 16.724502] random: crng init done [ 16.726016] random: 7 urandom warning(s) missed due to ratelimiting [ 16.771205] dracut-initqueue[580]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 19.020327] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Timers. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 22.412567] printk: systemd: 26 output lines suppressed due to ratelimiting [ 23.200855] SELinux: Disabled at runtime. [ 23.354641] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 23.370634] systemd[1]: Detected virtualization kvm. [ 23.372555] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 25.700308] systemd[1]: initrd-switch-root.service: Succeeded. [ 25.707369] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 25.730976] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 25.739618] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 25.751770] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 25.764317] systemd[1]: Starting Journal Service... Starting Journal Service... [ 25.777221] systemd[1]: Created slice User and Session Slice. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Listening on RPCbind Server Activation Socket. Mounting POSIX Message Queue File System... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Started Forward Password Requests to Wall Directory Watch. Starting Remount Root and Kernel File Systems... [ OK ] Created slice system-serial\x2dgetty.slice. Mounting Kernel Debug File System... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target rpc_pipefs.target. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Reached target RPC Port Mapper. Mounting Huge Pages File System... [ OK ] Created slice system-getty.slice. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. Starting Apply Kernel Variables... [ OK ] Listening on Process Core Dump Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. Activating swap /dev/disk/by-label/SWAP... [ 26.493798] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Reached target Local Encrypted Volumes. [ OK ] Started Journal Service. [ OK ] Mounted POSIX Message Queue File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted Huge Pages File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ 27.449712] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 29.055840] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 29.082074] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 29.691631] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 29.857420] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (8s / no limit) [** ] A start job is running for Configur…-only root support (8s / no limit) [*** ] A start job is running for Configur…-only root support (9s / no limit)[ 35.122215] Key type dns_resolver registered [ *** ] A start job is running for Configur…-only root support (9s / no limit) [ *** ] A start job is running for Configur…only root support (10s / no limit) [ ***] A start job is running for Configur…only root support (10s / no limit)[ 36.404872] NFS: Registering the id_resolver key type [ 36.413323] Key type id_resolver registered [ 36.423549] Key type id_legacy registered [ **] A start job is running for Configur…only root support (11s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Login Service... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started irqbalance daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. Starting Restore /run/initramfs on shutdown... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg149-client login: [ 85.990355] hrtimer: interrupt took 2526321 ns [ 114.736457] libcfs: loading out-of-tree module taints kernel. [ 114.940452] Key type ._llcrypt registered [ 114.949321] Key type .llcrypt registered [ 115.925502] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 116.006146] alg: No test for adler32 (adler32-zlib) [ 117.925516] Lustre: Lustre: Build Version: 2.17.57_1_g9da61ce [ 119.555520] LNet: Added LNI 192.168.201.49@tcp [8/256/0/180] [ 121.495633] Key type lgssc registered [ 123.135464] Lustre: Echo OBD driver; http://www.lustre.org/ [ 260.670385] Lustre: Mounted lustre-client - version 2.17.57_1_g9da61ce [ 266.635309] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 280.212815] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing check_logdir /tmp/testlogs/ [ 285.973834] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing yml_node [ 286.176208] Lustre: lustre-OST0000-osc-ffff8eade0694000: disconnect after 23s idle [ 290.586565] Lustre: DEBUG MARKER: Client: 2.17.57.1 [ 292.826404] Lustre: DEBUG MARKER: MDS: 2.17.57.1 [ 296.297580] Lustre: DEBUG MARKER: OSS: 2.17.57.1 [ 298.075831] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Wed Aug 19 01:39:59 EDT 2026 [ 313.926889] Lustre: Unmounted lustre-client [ 331.165705] Key type lgssc unregistered [ 331.448056] LNet: 8414:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 331.452502] LNetError: 8414:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 331.478223] LNet: Removed LNI 192.168.201.49@tcp [ 332.214194] Key type .llcrypt unregistered [ 332.218052] Key type ._llcrypt unregistered [ 359.309239] Key type ._llcrypt registered [ 359.314147] Key type .llcrypt registered [ 359.556092] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 359.590996] alg: No test for adler32 (adler32-zlib) [ 360.525100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 360.660303] LNet: Added LNI 192.168.201.49@tcp [8/256/0/180] [ 360.688979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 365.177858] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 365.258918] LNet: 9293:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 365.283300] LNetError: 9293:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 365.301356] LNet: Removed LNI 192.168.201.49@tcp [ 366.080464] Key type .llcrypt unregistered [ 366.081779] Key type ._llcrypt unregistered [ 372.686452] Key type ._llcrypt registered [ 372.689300] Key type .llcrypt registered [ 374.049317] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 01:41:15 (1787118075) [ 374.387823] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 374.405286] alg: No test for adler32 (adler32-zlib) [ 375.313369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 375.392356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 375.405474] LNet: 9717:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 375.415892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 375.443831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 375.474756] LNet: 9724:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 382.043331] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 01:41:24 (1787118084) [ 382.281161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 382.305143] LNet: 10302:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 382.317985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 382.405197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 389.879812] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 01:41:32 (1787118092) [ 390.600755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 390.637917] LNet: 10889:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 390.690538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 390.853784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 399.933323] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 01:41:41 (1787118101) [ 400.160894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 400.195794] LNet: 11468:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 400.246997] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 400.456163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 408.250391] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 01:41:49 (1787118109) [ 408.478398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 408.508901] LNet: 12058:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 408.534433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 408.618766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 408.994585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 416.990538] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 01:41:58 (1787118118) [ 417.300640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 417.325763] LNet: 12648:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 417.345641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 417.411963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 418.071602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 418.383498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 426.761773] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 01:42:08 (1787118128) [ 426.910162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 426.958299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 427.041334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 434.014771] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 01:42:16 (1787118136) [ 434.331041] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 434.349189] LNet: 13830:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 434.353919] LNet: 13830:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 434.369806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 434.424098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 434.479774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 434.930378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 434.957323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 435.260864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 435.304169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 435.605977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 435.658955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 436.071803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 436.111746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 436.479413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 436.526313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 436.996303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 437.047291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 437.548923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 437.575861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 438.142848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 438.174467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 438.524411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 438.548408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 438.891621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 438.953316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 439.403618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 439.434876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 439.881866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 439.921902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 447.006065] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 01:42:28 (1787118148) [ 447.441822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 447.516842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 447.650070] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 447.697503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 455.077599] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 01:42:37 (1787118157) [ 455.573905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 455.658345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 455.777200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 455.842396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 463.201549] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 01:42:44 (1787118164) [ 463.330966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 463.408711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 463.499680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 463.530523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 469.941788] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 01:42:51 (1787118171) [ 470.144546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 470.208312] LNet: 16267:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 470.218372] LNet: 16267:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 470.263147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 470.399356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 470.461361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 478.164606] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 01:43:00 (1787118180) [ 478.477904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 478.511801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 478.632989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 478.668795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 486.988995] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 01:43:08 (1787118188) [ 487.467966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 487.517285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 487.613889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 487.692515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 495.024801] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 01:43:16 (1787118196) [ 495.094722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 495.122441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 495.201635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 495.603913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 502.952818] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 01:43:24 (1787118204) [ 503.056785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 503.128275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 503.394746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 503.703454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 510.157454] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 01:43:32 (1787118212) [ 510.323949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 510.365857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 510.474827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 510.696302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 517.945295] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 01:43:39 (1787118219) [ 518.291075] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 518.346723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 518.890809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 519.268981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 525.525294] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 01:43:47 (1787118227) [ 525.807708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 525.849957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 525.907635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 525.939848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 532.475498] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 01:43:54 (1787118234) [ 532.642778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 532.733962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 532.852634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 532.914914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 539.437743] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 01:44:01 (1787118241) [ 539.581686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 539.648560] LNet: 21710:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 539.664313] LNet: 21710:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 8 previous similar messages [ 539.715425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 539.823655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 546.308609] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 01:44:07 (1787118247) [ 546.472613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 546.529488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 546.666415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 553.917674] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 01:44:15 (1787118255) [ 554.013252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 554.058138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 554.172370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 554.634123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 560.978441] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 01:44:22 (1787118262) [ 561.235995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 561.309498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 561.376577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 561.420401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 568.342991] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 01:44:30 (1787118270) [ 568.637297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 568.687274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 568.760540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 568.791806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 575.861737] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 01:44:37 (1787118277) [ 576.080540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 576.110418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 576.168665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 576.211494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 583.337073] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 01:44:45 (1787118285) [ 583.686721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 583.784394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 583.920205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 583.943515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 591.644775] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 01:44:53 (1787118293) [ 591.755578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 591.829469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 591.983607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 592.186602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 592.208271] LNetError: 25830:0:(peer.c:2362:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 592.261275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 592.300717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 599.094137] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 01:45:00 (1787118300) [ 599.244828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 599.301222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 599.403373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.BaleeoaPan [ 599.456335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.BaleeoaPan [ 599.498889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.BaleeoaPan [ 605.944767] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 01:45:07 (1787118307) [ 606.133814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 606.180768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 606.229927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 606.257955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 606.293719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 612.914647] Lustre: DEBUG MARKER: == sanity-lnet test 40: LNetAddPeer merges a pre-existing non-primary peer NID ========================================================== 01:45:14 (1787118314) [ 613.234143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 613.272150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 613.386610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 613.438729] LNet: Added LNI 192.168.201.49@tcp [8/256/0/180] [ 614.867475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.49@tcp [ 616.181670] Lustre: Lustre: Build Version: 2.17.57_1_g9da61ce [ 618.071290] Key type lgssc registered [ 619.092862] Lustre: Echo OBD driver; http://www.lustre.org/ [ 623.183212] LNetError: 27592:0:(lib-move.c:2158:lnet_handle_find_routed_path()) peer 192.168.201.49@tcp99 has no available nets [ 625.983918] Key type lgssc unregistered [ 626.370490] LNetError: 28702:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 626.394203] LNet: Removed LNI 192.168.201.49@tcp [ 627.120866] Key type .llcrypt unregistered [ 627.125398] Key type ._llcrypt unregistered [ 634.483388] Key type ._llcrypt registered [ 634.484681] Key type .llcrypt registered [ 635.879565] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 01:45:37 (1787118337) [ 636.331113] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 636.355730] alg: No test for adler32 (adler32-zlib) [ 637.395859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 637.486472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 640.587686] LNet: Message forwarding will be enabled in 2s [ 643.681176] LNet: Message forwarding enabled [ 644.062685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 644.076553] LNet: Message forwarding will stop in 2s to 6s [ 648.773948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 649.959863] LNet: Message forwarding disabled [ 651.482562] LNet: Message forwarding will be enabled in 2s [ 655.072114] LNet: Message forwarding enabled [ 658.746904] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 01:46:00 (1787118360) [ 658.958960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 658.983385] LNet: 30112:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 660.307760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 660.410502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 667.192897] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 01:46:09 (1787118369) [ 667.452240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 667.511793] LNet: 30694:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 667.552447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 667.613683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 667.645818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 667.681897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 667.708542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 667.734325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 667.766077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 667.808100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 667.843865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 667.877884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 667.912980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 667.954970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 668.033085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 668.104229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 668.155371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 668.216698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 668.275783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 668.338870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 668.435118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 668.487741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 668.510369] LNetError: 30737:0:(router.c:625:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 674.963250] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 01:46:16 (1787118376) [ 675.443451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 675.464215] LNet: 31308:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 675.493946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 675.606347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 683.138101] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 01:46:24 (1787118384) [ 685.564684] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg149-server [ 687.670663] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 01:46:29 (1787118389) [ 687.855131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 687.897483] LNet: 32234:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 687.932864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 688.010853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 688.041718] LNet: Added LNI 192.168.201.49@tcp [8/256/0/180] [ 689.103987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.201.[50-54]@tcp [ 689.140324] LNet: 32373:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 689.190397] LNetError: 32249:0:(lib-lnet.h:1279:lnet_set_route_aliveness()) route to tcp1 through 192.168.201.50@tcp has gone from up to down [ 689.237479] LNetError: 32243:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.201.51@tcp is being used as a gateway but routing feature is not turned on [ 690.220402] LNetError: 32243:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.201.51@tcp is being used as a gateway but routing feature is not turned on [ 690.232274] LNetError: 32243:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 7 previous similar messages [ 691.260584] LNetError: 32243:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.201.51@tcp is being used as a gateway but routing feature is not turned on [ 691.277729] LNetError: 32243:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 3 previous similar messages [ 693.328784] LNetError: 32243:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.201.53@tcp is being used as a gateway but routing feature is not turned on [ 693.342441] LNetError: 32243:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 7 previous similar messages [ 697.447869] LNetError: 32243:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.201.52@tcp is being used as a gateway but routing feature is not turned on [ 697.464747] LNetError: 32243:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 15 previous similar messages [ 697.700459] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 01:46:39 (1787118399) [ 700.028181] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg149-server [ 701.581761] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 01:46:43 (1787118403) [ 701.684037] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 701.718758] LNet: 33306:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 701.730795] LNetError: 33306:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 702.760349] LNet: Removed LNI 192.168.201.49@tcp [ 702.785136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 702.853354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 702.868821] LNet: Added LNI 192.168.201.49@tcp [8/256/0/180] [ 703.499361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.201.[50-55/2]@tcp [ 703.530497] LNet: 33427:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 703.545213] LNet: 33427:0:(router.c:729:lnet_add_route()) Skipped 4 previous similar messages [ 703.571581] LNetError: 33320:0:(lib-lnet.h:1279:lnet_set_route_aliveness()) route to tcp103 through 192.168.201.50@tcp has gone from up to down [ 703.579450] LNetError: 33320:0:(lib-lnet.h:1279:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 703.580100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.201.[50-55/2]@tcp [ 711.294734] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 01:46:52 (1787118412) [ 711.374283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 711.406188] LNet: 34013:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 711.415869] LNetError: 34013:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 711.437653] LNet: Removed LNI 192.168.201.49@tcp [ 711.472983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 711.603194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 711.652439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 711.695453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 711.804443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 711.867520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 711.920819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 711.977087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 712.024857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 712.059749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 712.109586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 712.176461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 712.218656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 712.282316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 712.365709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 712.420904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 712.484527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 712.570625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 712.600797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 712.647694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 712.730175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 712.803736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 712.920213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 713.106910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 713.149476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 713.222736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 713.329936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 713.381724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 713.490320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 713.579577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 713.623962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 713.678982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 713.750386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 713.803573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 713.872774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 721.578371] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 01:47:03 (1787118423) [ 723.677697] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg149-server [ 725.468356] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 01:47:07 (1787118427) [ 727.955788] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg149-server [ 730.226464] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 01:47:11 (1787118431) [ 730.355196] LNet: 35414:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 730.361898] LNet: 35414:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 11 previous similar messages [ 731.214785] Key type .llcrypt unregistered [ 731.216777] Key type ._llcrypt unregistered [ 739.105239] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 739.327833] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 739.335623] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 739.339699] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 741.981614] Key type ._llcrypt registered [ 741.986874] Key type .llcrypt registered [ 742.197269] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 742.208769] alg: No test for adler32 (adler32-zlib) [ 743.142350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 743.193195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 743.223122] LNet: Added LNI 192.168.201.49@tcp [8/256/0/180] [ 743.766714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 743.794841] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 746.053267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 746.071796] LNet: Removed LNI 10.1.2.3@tcp [ 746.187486] LNet: 36149:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 746.192081] LNetError: 36149:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 746.206590] LNet: Removed LNI 192.168.201.49@tcp [ 746.869373] Key type .llcrypt unregistered [ 746.872804] Key type ._llcrypt unregistered [ 760.514648] Key type ._llcrypt registered [ 760.517345] Key type .llcrypt registered [ 762.021536] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 01:47:43 (1787118463) [ 762.656794] Key type .llcrypt unregistered [ 762.659608] Key type ._llcrypt unregistered [ 769.109427] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 769.253344] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 769.256167] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 769.259871] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 772.053567] Key type ._llcrypt registered [ 772.057395] Key type .llcrypt registered [ 772.426961] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 772.471860] alg: No test for adler32 (adler32-zlib) [ 773.403038] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 773.462393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 773.530064] LNet: Added LNI 192.168.201.49@tcp [8/256/0/180] [ 774.296976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 774.335423] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 777.805495] LNet: 37654:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 777.825412] LNetError: 37654:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 777.836592] LNet: Removed LNI 10.1.2.3@tcp [ 777.851469] LNet: Removed LNI 192.168.201.49@tcp [ 779.034213] Key type .llcrypt unregistered [ 779.040309] Key type ._llcrypt unregistered [ 796.573945] Key type ._llcrypt registered [ 796.581937] Key type .llcrypt registered [ 798.179251] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 01:48:19 (1787118499) [ 798.898389] Key type .llcrypt unregistered [ 798.900181] Key type ._llcrypt unregistered [ 804.624263] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 804.890812] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 804.898457] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 804.904421] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 807.574461] Key type ._llcrypt registered [ 807.578390] Key type .llcrypt registered [ 807.886388] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 807.912166] alg: No test for adler32 (adler32-zlib) [ 808.907188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 809.032442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 809.077469] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 811.280673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 811.292070] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 813.704565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 813.737851] LNet: Removed LNI 10.1.2.3@tcp [ 813.764403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 813.792888] LNet: Removed LNI 10.1.2.31@tcp [ 813.805190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 813.839104] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 814.494302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 814.520246] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 815.765890] LNet: 39585:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 815.772678] LNetError: 39585:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg:0 not found [ 815.787927] LNet: Removed LNI 10.1.2.31@tcp [ 815.825133] LNet: Removed LNI 10.1.2.3@tcp [ 816.544994] Key type .llcrypt unregistered [ 816.547548] Key type ._llcrypt unregistered [ 828.698804] Key type ._llcrypt registered [ 828.705323] Key type .llcrypt registered [ 830.410438] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 01:48:51 (1787118531) [ 830.867463] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 830.897563] alg: No test for adler32 (adler32-zlib) [ 831.821874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 831.892966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 831.929631] LNet: Added LNI 192.168.201.49@tcp [8/256/0/180] [ 832.696206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 832.713704] LNet: 40542:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 832.718910] LNetError: 40542:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 832.732733] LNet: Removed LNI 192.168.201.49@tcp [ 832.752563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 832.805382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.201.49@tcp [ 832.835932] LNet: Added LNI 192.168.201.49@tcp [8/256/0/180] [ 833.592542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.201.49@tcp [ 833.628369] LNet: Removed LNI 192.168.201.49@tcp [ 833.698521] LNet: 40677:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 834.519509] Key type .llcrypt unregistered [ 834.522282] Key type ._llcrypt unregistered [ 849.055385] Key type ._llcrypt registered [ 849.060854] Key type .llcrypt registered [ 850.565793] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 01:49:12 (1787118552) [ 852.922617] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg149-server [ 855.094920] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 01:49:16 (1787118556) [ 855.818284] Key type .llcrypt unregistered [ 855.825344] Key type ._llcrypt unregistered [ 864.109179] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 864.229121] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 864.238549] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 864.255214] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 865.778989] Key type ._llcrypt registered [ 865.782191] Key type .llcrypt registered [ 866.003858] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 866.018451] alg: No test for adler32 (adler32-zlib) [ 866.967294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 867.080407] LNet: Added LNI 192.168.201.49@tcp1 [8/256/0/180] [ 868.178646] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 870.463310] LNet: 42508:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 870.469758] LNetError: 42508:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 870.475588] LNet: Removed LNI 192.168.201.49@tcp1 [ 870.487341] LNet: Removed LNI 10.1.2.3@tcp2 [ 871.272492] Key type .llcrypt unregistered [ 871.274937] Key type ._llcrypt unregistered [ 884.328565] Key type ._llcrypt registered [ 884.334701] Key type .llcrypt registered [ 885.937407] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 01:49:47 (1787118587) [ 886.514870] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 886.538878] alg: No test for adler32 (adler32-zlib) [ 887.425223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 887.468318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 887.484046] LNet: Added LNI 192.168.201.49@tcp [8/256/0/180] [ 888.410367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 888.427709] LNet: 43483:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 888.437110] LNetError: 43483:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 888.449118] LNet: Removed LNI 192.168.201.49@tcp [ 888.471270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 888.525521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 888.551716] LNet: Added LNI 192.168.201.49@tcp [8/256/0/180] [ 898.035228] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 01:49:59 (1787118599) [ 898.559194] LNet: 44075:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 898.574715] LNetError: 44075:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 898.603318] LNet: Removed LNI 192.168.201.49@tcp [ 899.221851] Key type .llcrypt unregistered [ 899.226723] Key type ._llcrypt unregistered [ 909.562377] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 909.834328] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 909.846328] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 909.854621] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 912.749526] Key type ._llcrypt registered [ 912.756532] Key type .llcrypt registered [ 913.027801] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 913.040988] alg: No test for adler32 (adler32-zlib) [ 914.015734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 914.087661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 914.150210] LNet: Added LNI 192.168.201.49@tcp [8/256/0/180] [ 915.212684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 915.252670] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 917.958828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 918.002912] LNet: 44824:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 918.013954] LNetError: 44824:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 918.024761] LNet: Removed LNI 10.1.2.3@tcp [ 918.033897] LNet: Removed LNI 192.168.201.49@tcp [ 918.067928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 918.166334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 918.197889] LNet: Added LNI 192.168.201.49@tcp [8/256/0/180] [ 918.201797] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 918.927927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 918.951913] LNet: 44851:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 918.973740] LNetError: 44851:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 918.980398] LNetError: 44851:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 918.985870] LNet: Removed LNI 10.1.2.3@tcp [ 918.998729] LNet: Removed LNI 192.168.201.49@tcp [ 919.048595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 919.169127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 919.225353] LNet: Added LNI 192.168.201.49@tcp [8/256/0/180] [ 919.234345] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 919.907258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 919.929412] LNet: Added LNI 192.168.201.49@tcp2 [8/256/0/180] [ 921.140721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 921.174137] LNet: 45014:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 921.186101] LNetError: 45014:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 921.199819] LNetError: 45014:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 921.215202] LNet: Removed LNI 10.1.2.3@tcp [ 921.225835] LNet: Removed LNI 192.168.201.49@tcp [ 921.254183] LNet: Removed LNI 192.168.201.49@tcp2 [ 921.317503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 921.391610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 921.442520] LNet: Added LNI 192.168.201.49@tcp [8/256/0/180] [ 921.459590] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 921.467391] LNet: Added LNI 192.168.201.49@tcp2 [8/256/0/180] [ 931.254094] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 01:50:32 (1787118632) [ 931.674614] LNet: 45612:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 931.682661] LNetError: 45612:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 931.696931] LNetError: 45612:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 931.717292] LNet: Removed LNI 10.1.2.3@tcp [ 931.722108] LNet: Removed LNI 192.168.201.49@tcp [ 931.753889] LNet: Removed LNI 192.168.201.49@tcp2 [ 932.755863] Key type .llcrypt unregistered [ 932.757623] Key type ._llcrypt unregistered [ 940.260215] Key type ._llcrypt registered [ 940.262486] Key type .llcrypt registered [ 940.573789] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 940.611144] alg: No test for adler32 (adler32-zlib) [ 941.603677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 941.674849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 941.710739] LNet: Added LNI 192.168.201.49@tcp [8/256/0/180] [ 942.718547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 942.770877] LNet: 46011:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 942.781776] LNetError: 46011:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 942.801159] LNet: Removed LNI 192.168.201.49@tcp [ 942.829171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 942.939078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 942.965358] LNet: Added LNI 192.168.201.49@tcp [8/256/0/180] [ 943.986647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 944.013562] LNet: 46041:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 944.021562] LNetError: 46041:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 944.039422] LNet: Removed LNI 192.168.201.49@tcp [ 944.065515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 944.199589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 944.237160] LNet: Added LNI 192.168.201.49@tcp [8/256/0/180] [ 944.959791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 945.020205] LNet: 46071:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 945.028567] LNetError: 46071:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 945.062287] LNet: Removed LNI 192.168.201.49@tcp [ 945.086924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 945.159508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 945.184493] LNet: Added LNI 192.168.201.49@tcp [8/256/0/180] [ 945.667623] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 945.740646] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 945.745710] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 945.750781] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 948.236296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 948.274136] LNet: 46150:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 948.280737] LNetError: 46150:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 948.291699] LNet: Removed LNI 192.168.201.49@tcp [ 948.304440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 948.405701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 948.455762] LNet: Added LNI 192.168.201.49@tcp [8/256/0/180] [ 956.460117] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 01:50:58 (1787118658) [ 956.661764] LNet: 46747:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 956.678284] LNetError: 46747:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 956.696794] LNet: Removed LNI 192.168.201.49@tcp [ 957.409840] Key type .llcrypt unregistered [ 957.411725] Key type ._llcrypt unregistered [ 964.816214] Key type ._llcrypt registered [ 964.818929] Key type .llcrypt registered [ 964.977701] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 964.995386] alg: No test for adler32 (adler32-zlib) [ 965.959380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 966.015064] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 966.058900] LNet: Added LNI 192.168.201.49@tcp [8/256/0/180] [ 966.940680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 966.992933] LNet: 47151:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 967.009098] LNetError: 47151:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 967.040816] LNet: Removed LNI 192.168.201.49@tcp [ 967.083847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 967.192849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 967.236218] LNet: Added LNI 192.168.201.49@tcp [8/256/0/180] [ 974.543450] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 01:51:16 (1787118676) [ 974.998903] LNet: 47744:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 975.007364] LNetError: 47744:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 975.031230] LNet: Removed LNI 192.168.201.49@tcp [ 975.818728] Key type .llcrypt unregistered [ 975.821787] Key type ._llcrypt unregistered [ 983.333468] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 983.458765] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 983.462414] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 983.469656] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 986.116114] Key type ._llcrypt registered [ 986.118197] Key type .llcrypt registered [ 986.407152] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 986.450680] alg: No test for adler32 (adler32-zlib) [ 987.478254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 987.569059] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 987.598599] LNet: Added LNI 192.168.201.49@tcp [8/256/0/180] [ 988.953686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 988.973698] LNet: 48199:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 988.987591] LNetError: 48199:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 989.020117] LNet: Removed LNI 192.168.201.49@tcp [ 989.077825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 989.158529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 989.190534] LNet: Added LNI 192.168.201.49@tcp [8/256/0/180] [ 996.828583] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 01:51:38 (1787118698) [ 996.989347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 997.038045] LNet: 48793:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 997.052762] LNetError: 48793:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 997.076873] LNet: Removed LNI 192.168.201.49@tcp [ 997.131877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 997.330773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 1004.365398] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 01:51:46 (1787118706) [ 1004.525699] LNet: 49372:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1005.976890] Key type .llcrypt unregistered [ 1005.993897] Key type ._llcrypt unregistered [ 1015.476557] Key type ._llcrypt registered [ 1015.478313] Key type .llcrypt registered [ 1015.904896] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1015.936613] alg: No test for adler32 (adler32-zlib) [ 1016.901131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1016.966610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1016.991433] LNet: Added LNI 192.168.201.49@tcp [8/256/0/180] [ 1018.287840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1018.336322] LNet: 49783:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1018.343393] LNetError: 49783:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1018.371728] LNet: Removed LNI 192.168.201.49@tcp [ 1018.440971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1018.549332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 1018.608711] LNet: Added LNI 192.168.201.49@tcp [8/256/0/180] [ 1026.448885] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 01:52:07 (1787118727) [ 1026.528207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1026.546026] LNet: 50370:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1026.551694] LNetError: 50370:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1026.565822] LNet: Removed LNI 192.168.201.49@tcp [ 1026.615736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1026.670764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 1033.779982] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 01:52:15 (1787118735) [ 1035.421565] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 1037.897496] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 01:52:19 (1787118739) [ 1039.565947] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 1042.296788] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 01:52:23 (1787118743) [ 1044.568664] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 1046.629687] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 01:52:28 (1787118748) [ 1048.502802] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 1050.584829] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 01:52:32 (1787118752) [ 1052.434176] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 1055.256608] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing ========================================================== 01:52:36 (1787118756) [ 1055.464095] LNet: 52422:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1056.618964] Key type .llcrypt unregistered [ 1056.622589] Key type ._llcrypt unregistered [ 1066.037919] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1066.556603] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1066.569710] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1066.589211] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1069.347496] Key type ._llcrypt registered [ 1069.349837] Key type .llcrypt registered [ 1069.562268] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1069.581912] alg: No test for adler32 (adler32-zlib) [ 1070.590400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1070.701074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1070.762747] LNet: Added LNI 192.168.201.49@tcp [8/512/0/180] [ 1070.768379] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 1070.771157] LNet: 52756:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1070.901575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1070.938196] LNet: 52775:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1070.952110] LNetError: 52775:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1070.959382] LNet: Removed LNI 10.1.2.3@tcp [ 1070.966524] LNetError: 52775:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1070.996828] LNet: Removed LNI 192.168.201.49@tcp [ 1071.053934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1071.211259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1071.250660] LNet: Added LNI 192.168.201.49@tcp [8/256/0/180] [ 1071.365564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1071.382229] LNet: 52799:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1071.390264] LNetError: 52799:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1071.423393] LNet: Removed LNI 192.168.201.49@tcp [ 1071.484618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1071.544238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1071.570955] LNet: Added LNI 192.168.201.49@tcp [8/256/0/180] [ 1071.579036] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1071.588867] LNet: 52807:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1071.700953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1071.725992] LNet: 52825:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1071.730962] LNetError: 52825:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1071.736398] LNet: Removed LNI 10.1.2.3@tcp [ 1071.738589] LNetError: 52825:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1071.747927] LNet: Removed LNI 192.168.201.49@tcp [ 1071.762460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1071.821153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1071.858534] LNet: Added LNI 192.168.201.49@tcp [8/256/0/180] [ 1071.867033] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1071.869112] LNet: 52833:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1072.154889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1072.203852] LNet: 52859:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1072.219148] LNetError: 52859:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1072.225564] LNet: Removed LNI 10.1.2.3@tcp [ 1072.230582] LNetError: 52859:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1072.252964] LNet: Removed LNI 192.168.201.49@tcp [ 1072.285201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1072.421553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1072.457793] LNet: Added LNI 192.168.201.49@tcp [8/256/0/182] [ 1072.470633] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 1072.654463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1072.696372] LNet: 52887:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1072.711259] LNetError: 52887:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1072.718788] LNet: Removed LNI 10.1.2.3@tcp [ 1072.723565] LNetError: 52887:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1072.750768] LNet: Removed LNI 192.168.201.49@tcp [ 1072.811645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1072.939927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1072.992914] LNet: Added LNI 192.168.201.49@tcp [8/256/0/182] [ 1072.995986] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 1072.997965] LNet: 52895:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1073.165988] LNet: Removed LNI 10.1.2.3@tcp [ 1073.193441] LNet: Removed LNI 192.168.201.49@tcp [ 1073.863026] Key type .llcrypt unregistered [ 1073.865814] Key type ._llcrypt unregistered [ 1086.285855] Key type ._llcrypt registered [ 1086.293680] Key type .llcrypt registered [ 1087.613529] Lustre: DEBUG MARKER: == sanity-lnet test 180: FastReg map error should not leak descriptors ========================================================== 01:53:09 (1787118789) [ 1088.927261] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_180 Need o2ib NETTYPE [ 1090.370677] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 01:53:12 (1787118792) [ 1090.948570] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1090.972564] alg: No test for adler32 (adler32-zlib) [ 1091.940189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1091.996701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1092.033078] LNet: Added LNI 192.168.201.49@tcp [8/256/0/180] [ 1092.591923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.201.49 [ 1092.606906] LNet: 54167:0:(config.c:1738:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1092.614434] LNet: Removed LNI 192.168.201.49@tcp [ 1098.487381] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 01:53:20 (1787118800) [ 1098.903431] LNet: 54737:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1098.919547] LNetError: 54737:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1098.953128] LNet: Removed LNI 192.168.201.49@tcp [ 1099.745832] Key type .llcrypt unregistered [ 1099.750374] Key type ._llcrypt unregistered [ 1106.965095] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1107.210585] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1110.067894] Key type ._llcrypt registered [ 1110.070640] Key type .llcrypt registered [ 1110.253912] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1110.270523] alg: No test for adler32 (adler32-zlib) [ 1111.567192] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1118.766285] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 01:53:40 (1787118820) [ 1118.964604] LNet: 55653:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1118.975276] LNetError: 55653:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1119.013313] LNet: Removed LNI 10.1.2.3@tcp [ 1120.069633] Key type .llcrypt unregistered [ 1120.075715] Key type ._llcrypt unregistered [ 1127.585741] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1127.889622] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1127.895240] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1127.902624] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1130.586996] Key type ._llcrypt registered [ 1130.588651] Key type .llcrypt registered [ 1130.703538] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1130.749857] alg: No test for adler32 (adler32-zlib) [ 1131.844827] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1139.641824] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 01:54:01 (1787118841) [ 1141.269705] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 1143.193706] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 01:54:05 (1787118845) [ 1143.430717] LNet: 56845:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1143.438043] LNetError: 56845:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1143.457567] LNet: Removed LNI 10.1.2.3@tcp [ 1144.215673] Key type .llcrypt unregistered [ 1144.221713] Key type ._llcrypt unregistered [ 1151.042963] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1151.303366] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1151.309583] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1151.317981] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1154.021491] Key type ._llcrypt registered [ 1154.023877] Key type .llcrypt registered [ 1154.271542] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1154.294910] alg: No test for adler32 (adler32-zlib) [ 1155.252980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1155.315637] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1162.960892] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 01:54:25 (1787118865) [ 1163.394689] LNet: 58078:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1163.401600] LNetError: 58078:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1163.416449] LNet: Removed LNI 10.1.2.3@tcp [ 1163.944656] Key type .llcrypt unregistered [ 1163.946407] Key type ._llcrypt unregistered [ 1169.923524] Key type ._llcrypt registered [ 1169.927731] Key type .llcrypt registered [ 1170.121792] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1170.153511] alg: No test for adler32 (adler32-zlib) [ 1171.097262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1171.127467] LNet: Added LNI 192.168.201.49@tcp [8/256/0/180] [ 1179.571524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.149@tcp [ 1180.422943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.49@tcp -d 192.168.201.149@tcp -m GET -e local_interrupt -r 1 [ 1180.476955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.49@tcp -d 192.168.201.49@tcp -m GET -e local_interrupt -r 1 [ 1180.535278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.149@tcp [ 1180.551886] LNet: There was an unexpected network error while writing to 192.168.201.149: rc = -22 [ 1180.812560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1180.844385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1180.894195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1180.944376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1180.982154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1181.025380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1181.060654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1181.104917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1181.137726] LNet: 58520:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1181.156687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1181.223441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1181.296453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1181.335543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1181.401997] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1181.407337] LNet: 1 local NIs in recovery (showing 1): 192.168.201.49@tcp [ 1181.408680] LNet: There was an unexpected network error while writing to 192.168.201.49: rc = -22 [ 1181.455430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1181.490301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1181.985427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.49@tcp -d 192.168.201.149@tcp -m GET -e local_dropped -r 1 [ 1182.014767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.49@tcp -d 192.168.201.49@tcp -m GET -e local_dropped -r 1 [ 1182.047897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.149@tcp [ 1182.257192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1182.285619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1182.315676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1182.342194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1182.374574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1182.398576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1182.432882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1182.460281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1182.476185] LNet: 58630:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1182.490447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1182.534351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1182.574272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1182.649562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1182.686818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1182.718467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1182.752983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1183.172197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.49@tcp -d 192.168.201.149@tcp -m GET -e local_aborted -r 1 [ 1183.212540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.49@tcp -d 192.168.201.49@tcp -m GET -e local_aborted -r 1 [ 1183.264473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.149@tcp [ 1183.280783] LNet: There was an unexpected network error while writing to 192.168.201.149: rc = -22 [ 1183.286063] LNet: Skipped 1 previous similar message [ 1183.527806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1183.552781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1183.577838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1183.602123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1183.633618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1183.666306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1183.706299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1183.763598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1183.783662] LNet: 58739:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1183.802496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1183.837192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1183.880545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1183.946332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1183.976304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1184.013482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1184.066050] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1184.513243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.49@tcp -d 192.168.201.149@tcp -m GET -e local_no_route -r 1 [ 1184.551396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.49@tcp -d 192.168.201.49@tcp -m GET -e local_no_route -r 1 [ 1184.597718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.149@tcp [ 1184.829688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1184.860434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1184.887753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1184.921313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1184.963388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1184.987549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1185.014108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1185.038476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1185.063365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1185.086329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1185.125878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1185.157894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1185.194623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1185.235469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1185.289542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1186.079801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.49@tcp -d 192.168.201.149@tcp -m GET -e local_timeout -r 1 [ 1186.133509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.49@tcp -d 192.168.201.49@tcp -m GET -e local_timeout -r 1 [ 1186.171292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.149@tcp [ 1186.197392] LNet: There was an unexpected network error while writing to 192.168.201.149: rc = -22 [ 1186.203327] LNet: Skipped 2 previous similar messages [ 1186.379800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1186.404950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1186.429613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1186.461349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1186.493409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1186.516391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1186.561608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1186.607680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1186.625684] LNet: 58957:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1186.630919] LNet: 58957:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1186.678076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1186.732547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1186.773648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1186.846826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1186.902176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1186.966981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1187.005311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1187.485515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.49@tcp -d 192.168.201.149@tcp -m GET -e local_error -r 1 [ 1187.514481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.49@tcp -d 192.168.201.49@tcp -m GET -e local_error -r 1 [ 1187.546613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.149@tcp [ 1187.742534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1187.770521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1187.795566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1187.818931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1187.847274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1187.869524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1187.904367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1187.926342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1187.956264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1187.984158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1188.014266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1188.036623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1188.071517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1188.093398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1188.118960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1188.386397] LNet: 59112:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1188.389902] LNetError: 59112:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1188.405496] LNet: Removed LNI 192.168.201.49@tcp [ 1188.819642] Key type .llcrypt unregistered [ 1188.822159] Key type ._llcrypt unregistered [ 1206.354610] Key type ._llcrypt registered [ 1206.358783] Key type .llcrypt registered [ 1207.505544] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 01:55:09 (1787118909) [ 1208.224454] Key type .llcrypt unregistered [ 1208.226648] Key type ._llcrypt unregistered [ 1215.321306] Key type ._llcrypt registered [ 1215.323535] Key type .llcrypt registered [ 1215.480666] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1215.497640] alg: No test for adler32 (adler32-zlib) [ 1216.426127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1216.448572] LNet: Added LNI 192.168.201.49@tcp [8/256/0/180] [ 1224.527550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.149@tcp [ 1238.188119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1238.298949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1238.336136] LNet: Added LNI 192.168.201.49@tcp1 [8/256/0/180] [ 1240.388336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.49@tcp -d 192.168.201.149@tcp -m GET -e local_interrupt -r 1 [ 1240.424425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.49@tcp -d 192.168.201.149@tcp1 -m GET -e local_interrupt -r 1 [ 1240.469457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.49@tcp -d 192.168.201.49@tcp -m GET -e local_interrupt -r 1 [ 1240.516068] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.49@tcp -d 192.168.201.49@tcp1 -m GET -e local_interrupt -r 1 [ 1240.568564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.49@tcp1 -d 192.168.201.149@tcp -m GET -e local_interrupt -r 1 [ 1240.615483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.49@tcp1 -d 192.168.201.149@tcp1 -m GET -e local_interrupt -r 1 [ 1240.663956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.49@tcp1 -d 192.168.201.49@tcp -m GET -e local_interrupt -r 1 [ 1240.695429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.49@tcp1 -d 192.168.201.49@tcp1 -m GET -e local_interrupt -r 1 [ 1240.730203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.149@tcp [ 1240.749579] LNet: There was an unexpected network error while writing to 192.168.201.149: rc = -22 [ 1240.759531] LNet: 1 local NIs in recovery (showing 1): 192.168.201.49@tcp [ 1241.021112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1241.064452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1241.134968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1241.216531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1241.288895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1241.324593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1241.366866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1241.402534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1241.421038] LNet: 60585:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1241.444129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1241.494438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1241.532576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1241.565432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1241.600126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1241.634826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1241.674529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1242.211273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.49@tcp -d 192.168.201.149@tcp -m GET -e local_dropped -r 1 [ 1242.242731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.49@tcp -d 192.168.201.149@tcp1 -m GET -e local_dropped -r 1 [ 1242.271415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.49@tcp -d 192.168.201.49@tcp -m GET -e local_dropped -r 1 [ 1242.303312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.49@tcp -d 192.168.201.49@tcp1 -m GET -e local_dropped -r 1 [ 1242.340622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.49@tcp1 -d 192.168.201.149@tcp -m GET -e local_dropped -r 1 [ 1242.377187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.49@tcp1 -d 192.168.201.149@tcp1 -m GET -e local_dropped -r 1 [ 1242.420439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.49@tcp1 -d 192.168.201.49@tcp -m GET -e local_dropped -r 1 [ 1242.455694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.49@tcp1 -d 192.168.201.49@tcp1 -m GET -e local_dropped -r 1 [ 1242.502920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.149@tcp [ 1242.517986] LNet: There was an unexpected network error while writing to 192.168.201.149: rc = -22 [ 1242.521231] LNet: Skipped 2 previous similar messages [ 1242.772569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1242.828215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1242.882288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1242.939903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1243.000865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1243.037430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1243.079882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1243.119905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1243.165235] LNet: 60707:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1243.203635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1243.248632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1243.295220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1243.326591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1243.377899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1243.428752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1243.489364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1244.201424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.49@tcp -d 192.168.201.149@tcp -m GET -e local_aborted -r 1 [ 1244.273898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.49@tcp -d 192.168.201.149@tcp1 -m GET -e local_aborted -r 1 [ 1244.351236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.49@tcp -d 192.168.201.49@tcp -m GET -e local_aborted -r 1 [ 1244.421962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.49@tcp -d 192.168.201.49@tcp1 -m GET -e local_aborted -r 1 [ 1244.511525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.49@tcp1 -d 192.168.201.149@tcp -m GET -e local_aborted -r 1 [ 1244.602911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.49@tcp1 -d 192.168.201.149@tcp1 -m GET -e local_aborted -r 1 [ 1244.695861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.49@tcp1 -d 192.168.201.49@tcp -m GET -e local_aborted -r 1 [ 1244.775256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.49@tcp1 -d 192.168.201.49@tcp1 -m GET -e local_aborted -r 1 [ 1244.843970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.149@tcp [ 1244.914378] LNet: There was an unexpected network error while writing to 192.168.201.149: rc = -22 [ 1244.921657] LNet: Skipped 2 previous similar messages [ 1245.453740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1245.502327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1245.561465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1245.642480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1245.703482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1245.774689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1245.847448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1245.917533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1245.957502] LNet: 60829:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1245.985406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1246.027905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1246.139738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1246.172221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1246.209402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1246.311372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1246.345916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1246.860668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.49@tcp -d 192.168.201.149@tcp -m GET -e local_no_route -r 1 [ 1246.897184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.49@tcp -d 192.168.201.149@tcp1 -m GET -e local_no_route -r 1 [ 1246.934882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.49@tcp -d 192.168.201.49@tcp -m GET -e local_no_route -r 1 [ 1247.006825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.49@tcp -d 192.168.201.49@tcp1 -m GET -e local_no_route -r 1 [ 1247.098493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.49@tcp1 -d 192.168.201.149@tcp -m GET -e local_no_route -r 1 [ 1247.131948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.49@tcp1 -d 192.168.201.149@tcp1 -m GET -e local_no_route -r 1 [ 1247.249657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.49@tcp1 -d 192.168.201.49@tcp -m GET -e local_no_route -r 1 [ 1247.309682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.49@tcp1 -d 192.168.201.49@tcp1 -m GET -e local_no_route -r 1 [ 1247.394114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.149@tcp [ 1247.419948] LNet: There was an unexpected network error while writing to 192.168.201.149: rc = -22 [ 1247.426796] LNet: Skipped 4 previous similar messages [ 1247.779617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1247.822818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1247.880902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1247.926738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1247.997830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1248.045021] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1248.097334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1248.148692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1248.174905] LNet: 60950:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1248.192594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1248.245893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1248.303462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1248.410329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1248.508839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1248.587444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1248.651485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1249.608833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.49@tcp -d 192.168.201.149@tcp -m GET -e local_timeout -r 1 [ 1249.672615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.49@tcp -d 192.168.201.149@tcp1 -m GET -e local_timeout -r 1 [ 1249.770275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.49@tcp -d 192.168.201.49@tcp -m GET -e local_timeout -r 1 [ 1249.846751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.49@tcp -d 192.168.201.49@tcp1 -m GET -e local_timeout -r 1 [ 1249.931979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.49@tcp1 -d 192.168.201.149@tcp -m GET -e local_timeout -r 1 [ 1250.016665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.49@tcp1 -d 192.168.201.149@tcp1 -m GET -e local_timeout -r 1 [ 1250.096993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.49@tcp1 -d 192.168.201.49@tcp -m GET -e local_timeout -r 1 [ 1250.177686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.49@tcp1 -d 192.168.201.49@tcp1 -m GET -e local_timeout -r 1 [ 1250.278264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.149@tcp [ 1250.758867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1250.822112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1250.895644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1250.967266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1251.031849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1251.132666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1251.235663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1251.290542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1251.367887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1251.447396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1251.505187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1251.542876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1251.590526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1251.639746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1251.679306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1252.387204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.49@tcp -d 192.168.201.149@tcp -m GET -e local_error -r 1 [ 1252.457791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.49@tcp -d 192.168.201.149@tcp1 -m GET -e local_error -r 1 [ 1252.516525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.49@tcp -d 192.168.201.49@tcp -m GET -e local_error -r 1 [ 1252.561328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.49@tcp -d 192.168.201.49@tcp1 -m GET -e local_error -r 1 [ 1252.623525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.49@tcp1 -d 192.168.201.149@tcp -m GET -e local_error -r 1 [ 1252.697129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.49@tcp1 -d 192.168.201.149@tcp1 -m GET -e local_error -r 1 [ 1252.745271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.49@tcp1 -d 192.168.201.49@tcp -m GET -e local_error -r 1 [ 1252.796859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.49@tcp1 -d 192.168.201.49@tcp1 -m GET -e local_error -r 1 [ 1252.863869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.149@tcp [ 1252.896167] LNet: There was an unexpected network error while writing to 192.168.201.149: rc = -22 [ 1252.911117] LNet: Skipped 5 previous similar messages [ 1253.260072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1253.298168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1253.356242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1253.411103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1253.502044] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1253.530794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1253.570836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1253.605470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1253.625171] LNet: 61193:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1253.638478] LNet: 61193:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1253.685507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1253.723785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1253.750916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1253.802203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1253.852701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1253.931952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1253.970538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1262.388339] LNet: 61274:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1262.397852] LNetError: 61274:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1262.409795] LNet: Removed LNI 192.168.201.49@tcp [ 1263.470176] LNet: Removed LNI 192.168.201.49@tcp1 [ 1263.936840] Key type .llcrypt unregistered [ 1263.941890] Key type ._llcrypt unregistered [ 1284.231630] Key type ._llcrypt registered [ 1284.234693] Key type .llcrypt registered [ 1285.852284] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 01:56:27 (1787118987) [ 1286.660477] Key type .llcrypt unregistered [ 1286.672765] Key type ._llcrypt unregistered [ 1293.925708] Key type ._llcrypt registered [ 1293.932017] Key type .llcrypt registered [ 1294.245493] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1294.286173] alg: No test for adler32 (adler32-zlib) [ 1295.309574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1295.356522] LNet: Added LNI 192.168.201.49@tcp [8/256/0/180] [ 1304.118307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.149@tcp [ 1304.995212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.49@tcp -d 192.168.201.149@tcp -m GET -e remote_dropped -r 1 [ 1305.031194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.49@tcp -d 192.168.201.49@tcp -m GET -e remote_dropped -r 1 [ 1305.082604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.149@tcp [ 1305.123463] LNet: There was an unexpected network error while writing to 192.168.201.149: rc = -22 [ 1305.549126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1305.608559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1305.673792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1305.742102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1305.796212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1305.839986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1305.884891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1305.931177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1305.969519] LNet: 62527:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1306.011235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1306.064276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1306.107438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1306.164271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1306.239292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1306.283911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1306.380391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1307.207885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.49@tcp -d 192.168.201.149@tcp -m GET -e remote_error -r 1 [ 1307.250961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.49@tcp -d 192.168.201.49@tcp -m GET -e remote_error -r 1 [ 1307.292348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.149@tcp [ 1307.322765] LNet: There was an unexpected network error while writing to 192.168.201.149: rc = -22 [ 1307.616183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1307.648794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1307.697451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1307.738193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1307.778183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1307.804523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1307.832900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1307.861610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1307.877730] LNet: 62636:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1307.906302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1307.945675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1307.993282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1308.036250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1308.072512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1308.106364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1308.136330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1308.828172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.49@tcp -d 192.168.201.149@tcp -m GET -e remote_timeout -r 1 [ 1308.859989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.49@tcp -d 192.168.201.49@tcp -m GET -e remote_timeout -r 1 [ 1308.889413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.149@tcp [ 1308.903030] LNet: There was an unexpected network error while writing to 192.168.201.149: rc = -22 [ 1309.225402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1309.267241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1309.332507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1309.378125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1309.406907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1309.453094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1309.496839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1309.570556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1309.599202] LNet: 62746:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1309.633238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1309.671305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1309.702830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1309.737455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1309.774316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1309.829887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1309.871204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1310.396594] LNet: 62791:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1310.404853] LNetError: 62791:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1310.420186] LNet: Removed LNI 192.168.201.49@tcp [ 1310.862558] Key type .llcrypt unregistered [ 1310.864498] Key type ._llcrypt unregistered [ 1329.102506] Key type ._llcrypt registered [ 1329.104713] Key type .llcrypt registered [ 1330.468959] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 01:57:12 (1787119032) [ 1331.473510] Key type .llcrypt unregistered [ 1331.475843] Key type ._llcrypt unregistered [ 1338.552910] Key type ._llcrypt registered [ 1338.555257] Key type .llcrypt registered [ 1338.716758] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1338.731824] alg: No test for adler32 (adler32-zlib) [ 1339.721174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1339.757588] LNet: Added LNI 192.168.201.49@tcp [8/256/0/180] [ 1347.865467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.149@tcp [ 1357.844539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1357.897958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1357.922643] LNet: Added LNI 192.168.201.49@tcp1 [8/256/0/180] [ 1359.460816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.49@tcp -d 192.168.201.149@tcp -m GET -e remote_dropped -r 1 [ 1359.496914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.49@tcp -d 192.168.201.149@tcp1 -m GET -e remote_dropped -r 1 [ 1359.527325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.49@tcp -d 192.168.201.49@tcp -m GET -e remote_dropped -r 1 [ 1359.553771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.49@tcp -d 192.168.201.49@tcp1 -m GET -e remote_dropped -r 1 [ 1359.602211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.49@tcp1 -d 192.168.201.149@tcp -m GET -e remote_dropped -r 1 [ 1359.658611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.49@tcp1 -d 192.168.201.149@tcp1 -m GET -e remote_dropped -r 1 [ 1359.710621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.49@tcp1 -d 192.168.201.49@tcp -m GET -e remote_dropped -r 1 [ 1359.778443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.49@tcp1 -d 192.168.201.49@tcp1 -m GET -e remote_dropped -r 1 [ 1359.812507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.149@tcp [ 1359.832600] LNet: There was an unexpected network error while writing to 192.168.201.149: rc = -22 [ 1359.837664] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.149@tcp [ 1360.064932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1360.091662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1360.115718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1360.138106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1360.162922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1360.197936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1360.230685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1360.258428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1360.272231] LNet: 64261:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1360.295522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1360.319970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1360.356314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1360.388472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1360.419764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1360.451395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1360.484508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1360.534666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1361.111525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.49@tcp -d 192.168.201.149@tcp -m GET -e remote_error -r 1 [ 1361.208201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.49@tcp -d 192.168.201.149@tcp1 -m GET -e remote_error -r 1 [ 1361.272088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.49@tcp -d 192.168.201.49@tcp -m GET -e remote_error -r 1 [ 1361.342671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.49@tcp -d 192.168.201.49@tcp1 -m GET -e remote_error -r 1 [ 1361.390754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.49@tcp1 -d 192.168.201.149@tcp -m GET -e remote_error -r 1 [ 1361.445890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.49@tcp1 -d 192.168.201.149@tcp1 -m GET -e remote_error -r 1 [ 1361.507116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.49@tcp1 -d 192.168.201.49@tcp -m GET -e remote_error -r 1 [ 1361.554871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.49@tcp1 -d 192.168.201.49@tcp1 -m GET -e remote_error -r 1 [ 1361.587633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.149@tcp [ 1361.606239] LNet: There was an unexpected network error while writing to 192.168.201.149: rc = -22 [ 1361.610629] LNet: Skipped 2 previous similar messages [ 1361.810544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1361.845525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1361.874259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1361.901968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1361.931226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1361.954477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1361.980995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1362.006687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1362.035301] LNet: 64384:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1362.054562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1362.092149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1362.123838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1362.165933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1362.199611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1362.234320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1362.272389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1362.304347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1362.847770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.49@tcp -d 192.168.201.149@tcp -m GET -e remote_timeout -r 1 [ 1362.876802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.49@tcp -d 192.168.201.149@tcp1 -m GET -e remote_timeout -r 1 [ 1362.906936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.49@tcp -d 192.168.201.49@tcp -m GET -e remote_timeout -r 1 [ 1362.943452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.49@tcp -d 192.168.201.49@tcp1 -m GET -e remote_timeout -r 1 [ 1362.968991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.49@tcp1 -d 192.168.201.149@tcp -m GET -e remote_timeout -r 1 [ 1363.001972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.49@tcp1 -d 192.168.201.149@tcp1 -m GET -e remote_timeout -r 1 [ 1363.033310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.49@tcp1 -d 192.168.201.49@tcp -m GET -e remote_timeout -r 1 [ 1363.074142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.49@tcp1 -d 192.168.201.49@tcp1 -m GET -e remote_timeout -r 1 [ 1363.109802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.149@tcp [ 1363.136829] LNet: There was an unexpected network error while writing to 192.168.201.149: rc = -22 [ 1363.423858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1363.476296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1363.552558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1363.601651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1363.638542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1363.670782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1363.722611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1363.759582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1363.778719] LNet: 64508:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1363.797912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1363.836051] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1363.876415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1363.921883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1363.967461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1364.004962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1364.038743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1364.098428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1370.010239] LNet: 64590:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1370.018653] LNetError: 64590:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1371.040225] LNet: Removed LNI 192.168.201.49@tcp [ 1371.047259] LNetError: 64590:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1371.065902] LNet: Removed LNI 192.168.201.49@tcp1 [ 1371.619555] Key type .llcrypt unregistered [ 1371.623313] Key type ._llcrypt unregistered [ 1389.543648] Key type ._llcrypt registered [ 1389.546588] Key type .llcrypt registered [ 1390.672401] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 01:58:12 (1787119092) [ 1392.889107] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 1395.037810] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 01:58:16 (1787119096) [ 1396.312382] Key type .llcrypt unregistered [ 1396.316870] Key type ._llcrypt unregistered [ 1404.109845] Key type ._llcrypt registered [ 1404.111515] Key type .llcrypt registered [ 1404.404692] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1404.430211] alg: No test for adler32 (adler32-zlib) [ 1405.369441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1405.398815] LNet: Added LNI 192.168.201.49@tcp [8/256/0/180] [ 1413.371968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.149@tcp [ 1414.115764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.49@tcp -d 192.168.201.149@tcp -m GET -e network_timeout -r 1 [ 1414.162573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.49@tcp -d 192.168.201.49@tcp -m GET -e network_timeout -r 1 [ 1414.233127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.149@tcp [ 1414.264419] LNet: There was an unexpected network error while writing to 192.168.201.149: rc = -22 [ 1414.517672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1414.542344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1414.562991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1414.596929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1414.623102] LNet: 1 local NIs in recovery (showing 1): 192.168.201.49@tcp [ 1414.633664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1414.684387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1414.712888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1414.757880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1414.792922] LNet: 66140:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1414.819527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1414.864588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1414.928438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1414.957589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1415.019371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1415.052894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1415.338257] LNet: 66183:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1415.345184] LNetError: 66183:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1415.357707] LNet: Removed LNI 192.168.201.49@tcp [ 1415.810489] Key type .llcrypt unregistered [ 1415.816214] Key type ._llcrypt unregistered [ 1434.718960] Key type ._llcrypt registered [ 1434.721960] Key type .llcrypt registered [ 1435.056924] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1435.081177] alg: No test for adler32 (adler32-zlib) [ 1436.290243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1436.370621] LNet: Added LNI 192.168.201.49@tcp [8/256/0/180] [ 1446.182736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.149@tcp [ 1459.094071] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1459.151617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1459.189287] LNet: Added LNI 192.168.201.49@tcp1 [8/256/0/180] [ 1460.718614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.49@tcp -d 192.168.201.149@tcp -m GET -e network_timeout -r 1 [ 1460.749218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.49@tcp -d 192.168.201.149@tcp1 -m GET -e network_timeout -r 1 [ 1460.778349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.49@tcp -d 192.168.201.49@tcp -m GET -e network_timeout -r 1 [ 1460.809402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.49@tcp -d 192.168.201.49@tcp1 -m GET -e network_timeout -r 1 [ 1460.836542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.49@tcp1 -d 192.168.201.149@tcp -m GET -e network_timeout -r 1 [ 1460.869253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.49@tcp1 -d 192.168.201.149@tcp1 -m GET -e network_timeout -r 1 [ 1460.897841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.49@tcp1 -d 192.168.201.49@tcp -m GET -e network_timeout -r 1 [ 1460.931578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.49@tcp1 -d 192.168.201.49@tcp1 -m GET -e network_timeout -r 1 [ 1460.970625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.149@tcp [ 1460.991549] LNet: There was an unexpected network error while writing to 192.168.201.149: rc = -22 [ 1461.191237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1461.215939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1461.243571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1461.266873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1461.296876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1461.321871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1461.345802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1461.371045] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1461.384384] LNet: 67067:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1461.400591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1461.426828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1461.457337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1461.487725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1461.512640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1461.544704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1465.981665] LNet: There was an unexpected network error while writing to 192.168.201.149: rc = -22 [ 1466.080289] LNet: 1 local NIs in recovery (showing 1): 192.168.201.49@tcp [ 1466.476273] LNet: 67145:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1466.484082] LNetError: 67145:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1466.495231] LNet: Removed LNI 192.168.201.49@tcp [ 1466.505573] LNet: Removed LNI 192.168.201.49@tcp1 [ 1466.912630] Key type .llcrypt unregistered [ 1466.919138] Key type ._llcrypt unregistered [ 1486.123221] Key type ._llcrypt registered [ 1486.127218] Key type .llcrypt registered [ 1487.861452] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 01:59:49 (1787119189) [ 1488.444842] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1488.467989] alg: No test for adler32 (adler32-zlib) [ 1489.486790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1489.534187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1489.584280] LNet: Added LNI 192.168.201.49@tcp [8/256/0/180] [ 1490.338635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1490.359369] LNet: Added LNI 192.168.201.49@tcp1 [8/256/0/180] [ 1491.187502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.49@tcp [ 1491.286785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1491.344368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1491.398103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1491.445336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1491.495411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1491.524928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.201.49@tcp [ 1491.541093] LNetError: 68246:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.201.49@tcp to recovery [ 1491.551769] LNet: 1 local NIs in recovery (showing 1): 192.168.201.49@tcp [ 1491.554947] LNet: There was an unexpected network error while writing to 192.168.201.49: rc = -22 [ 1493.599484] LNet: There was an unexpected network error while writing to 192.168.201.49: rc = -22 [ 1493.937989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1497.696198] LNet: There was an unexpected network error while writing to 192.168.201.49: rc = -22 [ 1498.767818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1498.821273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1498.896819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1498.910229] LNet: 68353:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1498.916331] LNetError: 68353:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1498.925680] LNet: Removed LNI 192.168.201.49@tcp [ 1498.940256] LNet: Removed LNI 192.168.201.49@tcp1 [ 1498.973923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1499.052148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1499.083930] LNet: Added LNI 192.168.201.49@tcp [8/256/0/180] [ 1500.196552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1500.249399] LNet: Added LNI 192.168.201.49@tcp1 [8/256/0/180] [ 1501.136747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.49@tcp [ 1501.222312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1501.318979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1501.405987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1501.477247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1501.552217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1501.602622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1501.652432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.201.49@tcp [ 1501.677450] LNetError: 68626:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.201.49@tcp to recovery [ 1502.114662] LNet: There was an unexpected network error while writing to 192.168.201.49: rc = -22 [ 1505.513242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1508.258490] LNet: There was an unexpected network error while writing to 192.168.201.49: rc = -22 [ 1508.268861] LNet: Skipped 1 previous similar message [ 1512.829396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1512.871622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1512.919717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1521.069526] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 02:00:22 (1787119222) [ 1521.554283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1521.581664] LNet: 69336:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1521.590217] LNetError: 69336:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1521.598310] LNetError: 69336:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1521.605873] LNet: Removed LNI 192.168.201.49@tcp [ 1521.619297] LNet: Removed LNI 192.168.201.49@tcp1 [ 1521.659759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1521.803032] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1521.877263] LNet: Added LNI 192.168.201.49@tcp [8/256/0/180] [ 1522.731882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1522.751076] LNet: Added LNI 192.168.201.49@tcp1 [8/256/0/180] [ 1523.714576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.49@tcp [ 1523.796054] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1523.820301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1523.856446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1523.894150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1523.930851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1523.982529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.201.49@tcp --health 0 [ 1524.151512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1524.767172] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.49@tcp [ 1525.791523] LNet: There was an unexpected network error while writing to 192.168.201.49: rc = -22 [ 1525.807658] LNet: Skipped 1 previous similar message [ 1540.291860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1540.540761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1540.643432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1540.678861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.201.49@tcp --health 500 [ 1540.759841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1546.305874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1546.369901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1546.391503] LNet: 69820:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1546.396832] LNetError: 69820:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1546.404201] LNetError: 69820:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1546.410730] LNet: Removed LNI 192.168.201.49@tcp [ 1546.424360] LNet: Removed LNI 192.168.201.49@tcp1 [ 1546.450994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1546.508686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1546.529735] LNet: Added LNI 192.168.201.49@tcp [8/256/0/180] [ 1547.132974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1547.153435] LNet: Added LNI 192.168.201.49@tcp1 [8/256/0/180] [ 1548.041660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.49@tcp [ 1548.068980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1548.099538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1548.121173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1548.159803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1548.207748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1548.241975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1548.266854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.201.49@tcp --health 0 [ 1549.599388] LNet: There was an unexpected network error while writing to 192.168.201.49: rc = -22 [ 1549.609499] LNet: Skipped 3 previous similar messages [ 1550.798964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1560.228667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1560.266444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1560.296475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1569.098649] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 02:01:11 (1787119271) [ 1569.729318] LNet: 70880:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1569.734492] LNetError: 70880:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1569.742784] LNetError: 70880:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1569.746970] LNet: Removed LNI 192.168.201.49@tcp [ 1569.753755] LNet: Removed LNI 192.168.201.49@tcp1 [ 1570.423987] Key type .llcrypt unregistered [ 1570.426578] Key type ._llcrypt unregistered [ 1577.890632] Key type ._llcrypt registered [ 1577.895331] Key type .llcrypt registered [ 1578.212615] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1578.227349] alg: No test for adler32 (adler32-zlib) [ 1579.147833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1579.207741] LNet: Added LNI 192.168.201.49@tcp [8/256/0/180] [ 1589.328340] Lustre: DEBUG MARKER: Initial discovery [ 1589.493572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.201.149@tcp [ 1591.918418] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1592.135406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1592.172285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.201.149@tcp [ 1592.189777] LNet: There was an unexpected network error while writing to 192.168.201.149: rc = -22 [ 1592.202404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1592.227748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.201.149@tcp -r 1 -m GET -l 3 [ 1592.256717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.201.149@tcp -r 1 -m GET -e local_error [ 1592.288652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.201.149@tcp -r 1 -m PUT -l 6 [ 1592.560488] LNet: 1 local NIs in recovery (showing 1): 192.168.201.49@tcp [ 1593.630865] Lustre: DEBUG MARKER: Force oleg149-client.virtnet to discover 192.168.201.149@tcp (in background) [ 1593.946246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.201.149@tcp [ 1595.388504] Lustre: DEBUG MARKER: Force oleg149-server to discover 192.168.201.49@tcp [ 1597.037573] LNet: There was an unexpected network error while writing to 192.168.201.149: rc = -22 [ 1598.030869] Lustre: DEBUG MARKER: Wait for 71648 [ 1599.738571] Lustre: DEBUG MARKER: Finished wait on 71648 [ 1600.207567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay del -a [ 1600.262612] LNet: There was an unexpected network error while writing to 192.168.201.149: rc = -22 [ 1600.277733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1600.560477] LNet: 72100:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1600.567822] LNetError: 72100:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1601.638895] LNet: Removed LNI 192.168.201.49@tcp [ 1602.239713] Key type .llcrypt unregistered [ 1602.243687] Key type ._llcrypt unregistered [ 1621.779699] Key type ._llcrypt registered [ 1621.782869] Key type .llcrypt registered [ 1623.148692] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 02:02:04 (1787119324) [ 1623.808515] Key type .llcrypt unregistered [ 1623.810609] Key type ._llcrypt unregistered [ 1631.216668] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1631.381488] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1631.390531] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1631.409728] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1634.291416] Key type ._llcrypt registered [ 1634.293750] Key type .llcrypt registered [ 1634.491647] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1634.506735] alg: No test for adler32 (adler32-zlib) [ 1635.481873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1635.548615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1635.575439] LNet: Added LNI 192.168.201.49@tcp [8/256/0/180] [ 1636.168517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1636.184905] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1645.976715] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 02:02:27 (1787119347) [ 1646.296228] LNet: 74183:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1646.305212] LNetError: 74183:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1646.313490] LNet: Removed LNI 10.1.2.3@tcp [ 1646.349235] LNet: Removed LNI 192.168.201.49@tcp [ 1646.915464] Key type .llcrypt unregistered [ 1646.917749] Key type ._llcrypt unregistered [ 1655.150935] Key type ._llcrypt registered [ 1655.152820] Key type .llcrypt registered [ 1655.278401] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1655.289991] alg: No test for adler32 (adler32-zlib) [ 1656.229437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1656.285819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1656.309424] LNet: Added LNI 192.168.201.49@tcp [8/256/0/180] [ 1657.070645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1657.095251] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1658.078200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.201.49@tcp [ 1659.168218] LNetError: Unexpected error -22 connecting to 192.168.201.49@tcp at host 192.168.201.49:988 [ 1664.779320] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 02:02:46 (1787119366) [ 1665.192175] LNet: 75320:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1665.198467] LNetError: 75320:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1665.202327] LNet: Removed LNI 10.1.2.3@tcp [ 1665.215150] LNet: Removed LNI 192.168.201.49@tcp [ 1666.110863] Key type .llcrypt unregistered [ 1666.119962] Key type ._llcrypt unregistered [ 1674.222893] Key type ._llcrypt registered [ 1674.227393] Key type .llcrypt registered [ 1674.497648] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1674.529985] alg: No test for adler32 (adler32-zlib) [ 1675.469717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1675.536940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1675.571829] LNet: Added LNI 192.168.201.49@tcp1 [8/256/0/180] [ 1676.308586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1676.328609] LNet: Added LNI 192.168.201.49@tcp2 [8/256/0/180] [ 1677.431620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.201.49@tcp1 --nid 192.168.201.49@tcp2 [ 1681.465677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.49@tcp1 192.168.201.49@tcp2 [ 1681.484642] LNetError: 76159:0:(lib-move.c:2101:lnet_handle_find_routed_path()) Unable to send message from 192.168.201.49@tcp1 to 192.168.201.49@tcp2 - Route table may be misconfigured [ 1681.502125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.49@tcp2 192.168.201.49@tcp1 [ 1688.316277] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 02:03:10 (1787119390) [ 1688.591900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1688.640968] LNet: 76731:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1688.653407] LNetError: 76731:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1689.695411] LNet: Removed LNI 192.168.201.49@tcp1 [ 1689.701358] LNetError: 76731:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1689.727082] LNet: Removed LNI 192.168.201.49@tcp2 [ 1689.776883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1689.847145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1689.893841] LNet: Added LNI 192.168.201.49@tcp [8/256/0/180] [ 1690.467500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1690.489718] LNet: Added LNI 192.168.201.49@tcp1 [8/256/0/180] [ 1691.440619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.49@tcp [ 1691.471708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.49@tcp 192.168.201.49@tcp [ 1691.537239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.49@tcp1 192.168.201.49@tcp1 [ 1691.614557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.201.49@tcp -d 192.168.201.49@tcp -e network_timeout [ 1691.665973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.201.49@tcp -d 192.168.201.49@tcp1 -e network_timeout [ 1691.723495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.201.49@tcp1 -d 192.168.201.49@tcp -e network_timeout [ 1691.777718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.201.49@tcp1 -d 192.168.201.49@tcp1 -e network_timeout [ 1691.849129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.49@tcp [ 1691.893788] LNet: There was an unexpected network error while writing to 192.168.201.49: rc = -22 [ 1691.933238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1692.032550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1692.175419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1692.896346] LNet: 1 local NIs in recovery (showing 1): 192.168.201.49@tcp [ 1700.092583] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 02:03:21 (1787119401) [ 1700.321704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1700.356544] LNet: 77595:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1700.363117] LNetError: 77595:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1701.407455] LNet: Removed LNI 192.168.201.49@tcp [ 1702.443737] LNet: Removed LNI 192.168.201.49@tcp1 [ 1702.497372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1702.567863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1702.900506] LNet: 77636:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1703.223741] Key type .llcrypt unregistered [ 1703.225726] Key type ._llcrypt unregistered [ 1715.766856] Key type ._llcrypt registered [ 1715.771656] Key type .llcrypt registered [ 1717.237894] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 02:03:38 (1787119418) [ 1717.800933] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1717.825492] alg: No test for adler32 (adler32-zlib) [ 1718.843323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1720.333702] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1722.276159] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 02:03:44 (1787119424) [ 1722.668351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1722.722808] LNet: 78735:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1722.769769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1722.887902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1722.923462] LNet: Added LNI 192.168.201.49@tcp [8/256/0/180] [ 1723.594781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1723.615368] LNet: Added LNI 192.168.201.49@tcp1 [8/256/0/180] [ 1724.758986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.49@tcp [ 1724.815936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.49@tcp1 [ 1724.887931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.49@tcp1 [ 1731.444876] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 02:03:53 (1787119433) [ 1734.000069] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg149-server [ 1736.020727] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 02:03:57 (1787119437) [ 1738.386367] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg149-server [ 1740.028756] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 02:04:02 (1787119442) [ 1742.073960] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg149-server [ 1743.780667] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 02:04:05 (1787119445) [ 1746.060165] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg149-server [ 1747.901364] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 02:04:09 (1787119449) [ 1750.061383] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg149-server [ 1752.230765] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 02:04:14 (1787119454) [ 1754.742873] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg149-server [ 1757.056237] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 02:04:18 (1787119458) [ 1759.539610] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg149-server [ 1762.059493] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 02:04:23 (1787119463) [ 1764.357498] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg149-server [ 1766.646513] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 02:04:28 (1787119468) [ 1768.956747] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg149-server [ 1771.656886] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 02:04:32 (1787119472) [ 1771.819232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1771.856831] LNet: 82608:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1771.863687] LNetError: 82608:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1771.871518] LNet: Removed LNI 192.168.201.49@tcp [ 1771.890555] LNet: Removed LNI 192.168.201.49@tcp1 [ 1771.931378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1772.058504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1772.093219] LNet: Added LNI 192.168.201.49@tcp [8/256/0/180] [ 1773.333694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1773.501986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.49@tcp [ 1773.792571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1773.829445] LNet: 82762:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1773.842526] LNetError: 82762:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1773.850120] LNetError: 82762:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1773.864923] LNet: Removed LNI 192.168.201.49@tcp [ 1773.911759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1774.038156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1774.139152] LNet: Added LNI 192.168.201.49@tcp [8/256/0/180] [ 1775.398734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.49@tcp [ 1775.682288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1775.732953] LNetError: 82908:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1776.741734] LNet: Removed LNI 192.168.201.49@tcp [ 1776.799827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1776.929557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1776.969909] LNet: Added LNI 192.168.201.49@tcp [8/256/0/180] [ 1778.089123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1778.213742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.49@tcp [ 1779.665820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1779.745920] LNet: 83064:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1779.761273] LNet: 83064:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1779.793779] LNetError: 83064:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1780.837982] LNet: Removed LNI 192.168.201.49@tcp [ 1780.895744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1780.967797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1780.992127] LNet: Added LNI 192.168.201.49@tcp [8/256/0/180] [ 1782.094975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.49@tcp [ 1782.345218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1783.400837] LNet: Removed LNI 192.168.201.49@tcp [ 1783.458827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1783.580189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1783.630062] LNet: Added LNI 192.168.201.49@tcp [8/256/0/180] [ 1784.393665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1784.461427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.49@tcp [ 1784.714977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1784.749345] LNet: 83357:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1784.756946] LNet: 83357:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1784.766327] LNetError: 83357:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1784.772844] LNetError: 83357:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1785.831458] LNet: Removed LNI 192.168.201.49@tcp [ 1785.870066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1785.951621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1786.001102] LNet: Added LNI 192.168.201.49@tcp [8/256/0/180] [ 1786.912799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.49@tcp [ 1787.273730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1788.330260] LNet: Removed LNI 192.168.201.49@tcp [ 1788.365630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1788.433777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1788.467828] LNet: Added LNI 192.168.201.49@tcp [8/256/0/180] [ 1789.290808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1789.383181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.49@tcp [ 1789.675770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1790.757867] LNet: Removed LNI 192.168.201.49@tcp [ 1790.797861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1790.872994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1790.894803] LNet: Added LNI 192.168.201.49@tcp [8/256/0/180] [ 1791.933892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.49@tcp [ 1792.287269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1793.393265] LNet: Removed LNI 192.168.201.49@tcp [ 1793.426942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1793.505768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1793.527944] LNet: Added LNI 192.168.201.49@tcp [8/256/0/180] [ 1794.292898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1794.353922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.49@tcp [ 1794.621824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1794.663521] LNet: 83943:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1794.668390] LNet: 83943:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 1794.680418] LNetError: 83943:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1794.688494] LNetError: 83943:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 1795.749943] LNet: Removed LNI 192.168.201.49@tcp [ 1795.827698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1795.914442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1795.967882] LNet: Added LNI 192.168.201.49@tcp [8/256/0/180] [ 1797.066155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.49@tcp [ 1797.300385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1798.374978] LNet: Removed LNI 192.168.201.49@tcp [ 1798.424306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1798.496590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1798.520173] LNet: Added LNI 192.168.201.49@tcp [8/256/0/180] [ 1799.212508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1799.279341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.49@tcp [ 1799.421771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1800.503366] LNet: Removed LNI 192.168.201.49@tcp [ 1800.556978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1800.629941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1800.665196] LNet: Added LNI 192.168.201.49@tcp [8/256/0/180] [ 1801.428155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.49@tcp [ 1802.767752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1803.827710] LNet: Removed LNI 192.168.201.49@tcp [ 1803.899305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1803.961470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1804.006906] LNet: Added LNI 192.168.201.49@tcp [8/256/0/180] [ 1804.595142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1804.637485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1804.660652] LNet: Removed LNI 192.168.201.49@tcp [ 1804.678328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1804.723685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1804.744437] LNet: Added LNI 192.168.201.49@tcp [8/256/0/180] [ 1805.398456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1805.457775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1805.478269] LNet: Removed LNI 192.168.201.49@tcp [ 1805.493750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1805.539052] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 128 [ 1805.597324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1805.637488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1805.683731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 1000000000000000000000000 [ 1805.758539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1805.792355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1805.846076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer foo [ 1805.906580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1805.946307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1806.004522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 8 [ 1806.028058] LNet: Added LNI 192.168.201.49@tcp [8/256/0/180] [ 1806.659866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1806.683922] LNet: Removed LNI 192.168.201.49@tcp [ 1806.695880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1806.741418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1813.340778] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 02:05:15 (1787119515) [ 1813.599784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1813.632692] LNet: 85415:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1813.648115] LNet: 85415:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 9 previous similar messages [ 1813.666140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1813.729472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1813.764839] LNet: Added LNI 192.168.201.49@tcp231 [8/256/0/180] [ 1815.480076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1815.524507] LNetError: 85553:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1815.538642] LNetError: 85553:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 6 previous similar messages [ 1815.566600] LNet: Removed LNI 192.168.201.49@tcp231 [ 1815.603494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1815.680259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1815.730101] LNet: Added LNI 192.168.201.49@tcp231 [8/256/0/180] [ 1815.734576] LNet: 85561:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1816.450325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1816.499798] LNet: Removed LNI 192.168.201.49@tcp231 [ 1816.517413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1816.560124] LNet: Added LNI 192.168.201.49@tcp231 [8/256/0/180] [ 1817.642533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1817.681237] LNet: Removed LNI 192.168.201.49@tcp231 [ 1817.703469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1817.777383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1817.798766] LNet: Added LNI 192.168.201.49@tcp231 [8/256/0/0] [ 1817.803262] LNet: 85714:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1818.209193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1818.242793] LNet: Removed LNI 192.168.201.49@tcp231 [ 1818.259453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1818.280180] LNet: Added LNI 192.168.201.49@tcp231 [8/256/0/0] [ 1819.361720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1819.395675] LNet: Removed LNI 192.168.201.49@tcp231 [ 1819.425288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1819.497502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1819.518462] LNet: Added LNI 192.168.201.49@tcp231 [8/256/0/60] [ 1819.522321] LNet: 85866:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1819.790515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1819.820654] LNet: Removed LNI 192.168.201.49@tcp231 [ 1819.834906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1819.863024] LNet: Added LNI 192.168.201.49@tcp231 [8/256/0/60] [ 1821.538759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1821.587222] LNet: Removed LNI 192.168.201.49@tcp231 [ 1821.626413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1821.775804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1821.821140] LNet: Added LNI 192.168.201.49@tcp231 [8/256/0/180] [ 1821.834396] LNet: 86017:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1822.515400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1822.554380] LNet: Removed LNI 192.168.201.49@tcp231 [ 1822.571811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1822.603077] LNet: Added LNI 192.168.201.49@tcp231 [8/256/0/180] [ 1831.764479] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 02:05:33 (1787119533) [ 1831.864345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1831.922180] LNet: Removed LNI 192.168.201.49@tcp231 [ 1831.962862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1832.100944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1832.138856] LNet: Added LNI 192.168.201.49@tcp232 [8/256/0/180] [ 1833.311922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1833.353225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1833.386818] LNet: Removed LNI 192.168.201.49@tcp232 [ 1833.416574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1833.437247] LNet: Added LNI 192.168.201.49@tcp232 [8/256/0/180] [ 1834.293432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1834.326366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1834.356402] LNet: Removed LNI 192.168.201.49@tcp232 [ 1834.370548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1834.396735] LNet: Added LNI 192.168.201.49@tcp232 [8/256/0/180] [ 1835.200945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1835.266401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1835.304523] LNet: Removed LNI 192.168.201.49@tcp232 [ 1835.326712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1835.355556] LNet: Added LNI 192.168.201.49@tcp232 [8/256/0/180] [ 1836.079866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1836.114415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1836.144840] LNet: Removed LNI 192.168.201.49@tcp232 [ 1836.180426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1842.945688] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 02:05:44 (1787119544) [ 1844.561704] Key type .llcrypt unregistered [ 1844.565217] Key type ._llcrypt unregistered [ 1852.459583] Key type ._llcrypt registered [ 1852.461668] Key type .llcrypt registered [ 1852.765611] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1852.789183] alg: No test for adler32 (adler32-zlib) [ 1853.879817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1853.942975] LNet: Added LNI 192.168.201.49@tcp [8/256/0/180] [ 1863.923329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.149@tcp [ 1871.881706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1871.913812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1871.931561] LNet: Added LNI 192.168.201.49@tcp1 [8/256/0/180] [ 1873.348944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.49@tcp -d 192.168.201.149@tcp -m GET -e local_interrupt -r 2 [ 1873.376804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.49@tcp -d 192.168.201.149@tcp1 -m GET -e local_interrupt -r 2 [ 1873.450903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.49@tcp -d 192.168.201.49@tcp -m GET -e local_interrupt -r 2 [ 1873.520250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.49@tcp -d 192.168.201.49@tcp1 -m GET -e local_interrupt -r 2 [ 1873.594212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.49@tcp1 -d 192.168.201.149@tcp -m GET -e local_interrupt -r 2 [ 1873.648774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.49@tcp1 -d 192.168.201.149@tcp1 -m GET -e local_interrupt -r 2 [ 1873.688683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.49@tcp1 -d 192.168.201.49@tcp -m GET -e local_interrupt -r 2 [ 1873.722410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.49@tcp1 -d 192.168.201.49@tcp1 -m GET -e local_interrupt -r 2 [ 1873.759566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.49@tcp 192.168.201.149@tcp [ 1873.783431] LNet: There was an unexpected network error while writing to 192.168.201.149: rc = -22 [ 1873.791671] LNet: 1 local NIs in recovery (showing 1): 192.168.201.49@tcp [ 1874.094087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1874.132904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1874.174257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1874.237860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1874.270103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1874.297324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1874.337346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1874.387623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1874.429469] LNet: 88524:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1874.458379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1874.499646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1874.537696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1874.563748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1874.589226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1874.608115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1875.048038] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.49@tcp -d 192.168.201.149@tcp -m GET -e local_dropped -r 2 [ 1875.101047] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.49@tcp -d 192.168.201.149@tcp1 -m GET -e local_dropped -r 2 [ 1875.132610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.49@tcp -d 192.168.201.49@tcp -m GET -e local_dropped -r 2 [ 1875.165888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.49@tcp -d 192.168.201.49@tcp1 -m GET -e local_dropped -r 2 [ 1875.195451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.49@tcp1 -d 192.168.201.149@tcp -m GET -e local_dropped -r 2 [ 1875.225257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.49@tcp1 -d 192.168.201.149@tcp1 -m GET -e local_dropped -r 2 [ 1875.256457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.49@tcp1 -d 192.168.201.49@tcp -m GET -e local_dropped -r 2 [ 1875.297438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.49@tcp1 -d 192.168.201.49@tcp1 -m GET -e local_dropped -r 2 [ 1875.331341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.49@tcp 192.168.201.149@tcp [ 1875.358329] LNet: There was an unexpected network error while writing to 192.168.201.149: rc = -22 [ 1875.599942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1875.659268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1875.708469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1875.765612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1875.806034] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1875.847883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1875.885857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1875.920765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1875.940954] LNet: 88644:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1875.964111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1875.997668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1876.034355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1876.065084] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1876.102607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1876.144465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1876.768896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.49@tcp -d 192.168.201.149@tcp -m GET -e local_aborted -r 2 [ 1876.824564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.49@tcp -d 192.168.201.149@tcp1 -m GET -e local_aborted -r 2 [ 1876.881500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.49@tcp -d 192.168.201.49@tcp -m GET -e local_aborted -r 2 [ 1876.929688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.49@tcp -d 192.168.201.49@tcp1 -m GET -e local_aborted -r 2 [ 1876.975443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.49@tcp1 -d 192.168.201.149@tcp -m GET -e local_aborted -r 2 [ 1877.035724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.49@tcp1 -d 192.168.201.149@tcp1 -m GET -e local_aborted -r 2 [ 1877.078483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.49@tcp1 -d 192.168.201.49@tcp -m GET -e local_aborted -r 2 [ 1877.109518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.49@tcp1 -d 192.168.201.49@tcp1 -m GET -e local_aborted -r 2 [ 1877.146795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.49@tcp 192.168.201.149@tcp [ 1877.177282] LNet: There was an unexpected network error while writing to 192.168.201.149: rc = -22 [ 1877.405914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1877.437397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1877.465413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1877.497665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1877.546877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1877.582820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1877.609853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1877.661179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1877.680560] LNet: 88764:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1877.695404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1877.726168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1877.756359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1877.824774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1877.873563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1877.907692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1878.390093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.49@tcp -d 192.168.201.149@tcp -m GET -e local_no_route -r 2 [ 1878.419548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.49@tcp -d 192.168.201.149@tcp1 -m GET -e local_no_route -r 2 [ 1878.445821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.49@tcp -d 192.168.201.49@tcp -m GET -e local_no_route -r 2 [ 1878.477379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.49@tcp -d 192.168.201.49@tcp1 -m GET -e local_no_route -r 2 [ 1878.515816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.49@tcp1 -d 192.168.201.149@tcp -m GET -e local_no_route -r 2 [ 1878.548766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.49@tcp1 -d 192.168.201.149@tcp1 -m GET -e local_no_route -r 2 [ 1878.577494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.49@tcp1 -d 192.168.201.49@tcp -m GET -e local_no_route -r 2 [ 1878.608511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.49@tcp1 -d 192.168.201.49@tcp1 -m GET -e local_no_route -r 2 [ 1878.643769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.49@tcp 192.168.201.149@tcp [ 1878.941975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1878.970031] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1878.996375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1879.025772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1879.051695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1879.078757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1879.112164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1879.142747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1879.167661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1879.205867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1879.259553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1879.290780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1879.322665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1879.362926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1879.884943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.49@tcp -d 192.168.201.149@tcp -m GET -e local_timeout -r 2 [ 1879.916674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.49@tcp -d 192.168.201.149@tcp1 -m GET -e local_timeout -r 2 [ 1879.943264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.49@tcp -d 192.168.201.49@tcp -m GET -e local_timeout -r 2 [ 1879.975512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.49@tcp -d 192.168.201.49@tcp1 -m GET -e local_timeout -r 2 [ 1880.003701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.49@tcp1 -d 192.168.201.149@tcp -m GET -e local_timeout -r 2 [ 1880.036130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.49@tcp1 -d 192.168.201.149@tcp1 -m GET -e local_timeout -r 2 [ 1880.088536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.49@tcp1 -d 192.168.201.49@tcp -m GET -e local_timeout -r 2 [ 1880.147083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.49@tcp1 -d 192.168.201.49@tcp1 -m GET -e local_timeout -r 2 [ 1880.205318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.49@tcp 192.168.201.149@tcp [ 1880.231959] LNet: There was an unexpected network error while writing to 192.168.201.149: rc = -22 [ 1880.238435] LNet: Skipped 1 previous similar message [ 1880.512378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1880.549710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1880.582377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1880.609291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1880.636546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1880.666717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1880.697735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1880.728986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1880.756834] LNet: 89005:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1880.763429] LNet: 89005:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1880.784296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1880.821669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1880.870228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1880.904623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1880.930464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1880.956265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1881.376358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.49@tcp -d 192.168.201.149@tcp -m GET -e remote_dropped -r 2 [ 1881.405724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.49@tcp -d 192.168.201.149@tcp1 -m GET -e remote_dropped -r 2 [ 1881.440377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.49@tcp -d 192.168.201.49@tcp -m GET -e remote_dropped -r 2 [ 1881.506701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.49@tcp -d 192.168.201.49@tcp1 -m GET -e remote_dropped -r 2 [ 1881.583538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.49@tcp1 -d 192.168.201.149@tcp -m GET -e remote_dropped -r 2 [ 1881.627267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.49@tcp1 -d 192.168.201.149@tcp1 -m GET -e remote_dropped -r 2 [ 1881.674298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.49@tcp1 -d 192.168.201.49@tcp -m GET -e remote_dropped -r 2 [ 1881.753390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.49@tcp1 -d 192.168.201.49@tcp1 -m GET -e remote_dropped -r 2 [ 1881.795652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.49@tcp 192.168.201.149@tcp [ 1881.841306] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.149@tcp [ 1882.065073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1882.094747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1882.121296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1882.144292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1882.189047] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1882.224235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1882.271967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1882.326571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1882.365947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1882.395602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1882.429434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1882.474200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1882.519284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1882.550445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1887.979794] LNet: 89203:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1887.990060] LNetError: 89203:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1888.007558] LNet: Removed LNI 192.168.201.49@tcp [ 1888.020728] LNet: Removed LNI 192.168.201.49@tcp1 [ 1888.622780] Key type .llcrypt unregistered [ 1888.625296] Key type ._llcrypt unregistered [ 1907.109536] Key type ._llcrypt registered [ 1907.111867] Key type .llcrypt registered [ 1909.104545] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 02:06:50 (1787119610) [ 1909.694164] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1909.713602] alg: No test for adler32 (adler32-zlib) [ 1910.671728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1910.724765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1910.743863] LNet: Added LNI 192.168.201.49@tcp [8/256/0/180] [ 1911.824491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1911.853506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.201.49@tcp [ 1911.869957] LNetError: 90184:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.201.49@tcp to recovery [ 1911.982723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.201.49@tcp [ 1912.101121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1918.249872] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 02:07:00 (1787119620) [ 1918.589925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1918.654939] LNet: 90773:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1918.663304] LNetError: 90773:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1918.690483] LNet: Removed LNI 192.168.201.49@tcp [ 1918.722659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1918.795143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1918.829691] LNet: Added LNI 192.168.201.49@tcp [8/256/0/180] [ 1919.554082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1919.572654] LNet: Added LNI 192.168.201.49@tcp1 [8/256/0/180] [ 1920.530589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.201.49@tcp [ 1920.555661] LNetError: 91039:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.201.49@tcp to recovery [ 1920.799963] LNet: 1 local NIs in recovery (showing 1): 192.168.201.49@tcp [ 1920.814816] LNet: There was an unexpected network error while writing to 192.168.201.49: rc = -22 [ 1921.790774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.201.49@tcp [ 1929.529354] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 02:07:11 (1787119631) [ 1931.370577] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg149-server [ 1933.254937] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 02:07:15 (1787119635) [ 1935.380613] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg149-server [ 1937.368372] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 02:07:19 (1787119639) [ 1939.238867] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg149-server [ 1940.977525] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 02:07:22 (1787119642) [ 1941.186425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1941.208593] LNet: 92661:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1941.214764] LNetError: 92661:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1942.239238] LNet: Removed LNI 192.168.201.49@tcp [ 1942.243253] LNetError: 92661:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1942.268305] LNet: Removed LNI 192.168.201.49@tcp1 [ 1942.316021] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1942.392110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1942.439281] LNet: Added LNI 192.168.201.49@tcp [8/256/0/180] [ 1943.252152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1943.353964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1943.429639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1943.543070] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1943.603664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1943.663857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1949.317779] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 02:07:31 (1787119651) [ 1949.490230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1949.505131] LNet: 93386:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1949.510137] LNetError: 93386:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1949.520270] LNet: Removed LNI 192.168.201.49@tcp [ 1949.544528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1949.584302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1949.606026] LNet: Added LNI 192.168.201.49@tcp [8/256/0/180] [ 1958.026143] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 02:07:39 (1787119659) [ 1958.405801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1958.435818] LNet: 94090:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1958.443155] LNetError: 94090:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1958.469275] LNet: Removed LNI 192.168.201.49@tcp [ 1958.511448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1958.574986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1958.611706] LNet: Added LNI 192.168.201.49@tcp [8/256/0/180] [ 1967.561632] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 02:07:49 (1787119669) [ 1969.423391] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 1971.032116] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 02:07:53 (1787119673) [ 1971.686513] LNet: 95125:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1971.700876] LNetError: 95125:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1971.742709] LNet: Removed LNI 192.168.201.49@tcp [ 1972.934083] Key type .llcrypt unregistered [ 1972.936149] Key type ._llcrypt unregistered [ 1979.576391] Key type ._llcrypt registered [ 1979.580345] Key type .llcrypt registered [ 1979.764473] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1979.801415] alg: No test for adler32 (adler32-zlib) [ 1980.766421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1980.787895] LNet: Added LNI 192.168.201.49@tcp [8/256/0/180] [ 1989.377348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.149@tcp [ 1996.393388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.201.149@tcp [ 1996.698857] LNet: 95550:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1996.703722] LNetError: 95550:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1996.727600] LNet: Removed LNI 192.168.201.49@tcp [ 1997.100721] Key type .llcrypt unregistered [ 1997.102780] Key type ._llcrypt unregistered [ 2009.608344] Key type ._llcrypt registered [ 2009.611300] Key type .llcrypt registered [ 2010.778353] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 2012.176534] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 2013.621041] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 02:08:35 (1787119715) [ 2015.275595] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 [ 2017.003426] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 02:08:38 (1787119718) [ 2018.569157] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg149-server [ 2020.465493] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 02:08:42 (1787119722) [ 2022.485374] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg149-server [ 2024.661211] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 02:08:46 (1787119726) [ 2026.297864] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2026.314080] alg: No test for adler32 (adler32-zlib) [ 2027.328590] Lustre: Lustre: Build Version: 2.17.57_1_g9da61ce [ 2027.607520] LNet: Added LNI 192.168.201.49@tcp [8/256/0/180] [ 2029.383176] Key type lgssc registered [ 2030.685963] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2053.301089] Key type lgssc unregistered [ 2053.532127] LNet: 98740:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2053.544363] LNetError: 98740:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2053.577408] LNet: Removed LNI 192.168.201.49@tcp [ 2054.488584] Key type .llcrypt unregistered [ 2054.493735] Key type ._llcrypt unregistered [ 2071.896440] Key type ._llcrypt registered [ 2071.907961] Key type .llcrypt registered [ 2073.341888] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 02:09:35 (1787119775) [ 2073.798509] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2073.809767] alg: No test for adler32 (adler32-zlib) [ 2074.780753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2074.860660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2074.917877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2074.952532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2075.022665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2082.939242] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 02:09:44 (1787119784) [ 2083.135323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2083.167827] LNet: 100155:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2083.186894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2083.247274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2083.277083] LNet: Added LNI 192.168.201.49@tcp [8/256/0/180] [ 2084.290702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2084.305720] LNet: 100300:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2084.310651] LNetError: 100300:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2084.326194] LNet: Removed LNI 192.168.201.49@tcp [ 2084.346392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2084.426786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 2084.447785] LNet: Added LNI 192.168.201.49@tcp [8/256/0/180] [ 2085.052227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2085.104834] LNetError: 100328:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2085.128650] LNet: Removed LNI 192.168.201.49@tcp [ 2085.147660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2085.218958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 2085.249776] LNet: Added LNI 192.168.201.49@tcp [8/256/0/180] [ 2092.943121] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 02:09:54 (1787119794) [ 2093.054963] LNet: 100909:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2093.062713] LNet: 100909:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 2093.070829] LNetError: 100909:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2093.099763] LNet: Removed LNI 192.168.201.49@tcp [ 2093.949699] Key type .llcrypt unregistered [ 2093.956505] Key type ._llcrypt unregistered [ 2101.892636] Key type ._llcrypt registered [ 2101.894897] Key type .llcrypt registered [ 2102.165628] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2102.199032] alg: No test for adler32 (adler32-zlib) [ 2103.168189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2103.252821] LNet: Added LNI 192.168.201.49@tcp [8/256/0/180] [ 2103.256508] LNet: *** cfs_fail_loc=e003, val=100*** [ 2103.903727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.49@tcp [ 2103.919627] LNet: *** cfs_fail_loc=e003, val=100*** [ 2103.921879] LNet: Skipped 2 previous similar messages [ 2103.926289] Ping reply received with mapping: [ 2103.928958] found metadata reply NI[192.168.201.49@tcp] [ 2103.932366] 31 c9 a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 2103.934764] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 2103.936782] 5a 5a 5a 5a 5a 5a 5a 5a [ 2104.033656] LNet: 101320:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2104.039929] LNetError: 101320:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2104.058305] LNet: Removed LNI 192.168.201.49@tcp [ 2104.721645] Key type .llcrypt unregistered [ 2104.724331] Key type ._llcrypt unregistered [ 2117.838558] Key type ._llcrypt registered [ 2117.841285] Key type .llcrypt registered [ 2119.726544] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 02:10:20 (1787119820) [ 2120.484506] Key type .llcrypt unregistered [ 2120.486561] Key type ._llcrypt unregistered [ 2128.883031] Key type ._llcrypt registered [ 2128.886745] Key type .llcrypt registered [ 2129.143396] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2129.167306] alg: No test for adler32 (adler32-zlib) [ 2130.502373] LNetError: 102377:0:(api-ni.c:2850:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 2130.916765] Key type .llcrypt unregistered [ 2130.920377] Key type ._llcrypt unregistered [ 2137.525185] Key type ._llcrypt registered [ 2137.526296] Key type .llcrypt registered [ 2138.659129] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 02:10:40 (1787119840) [ 2139.385537] Key type .llcrypt unregistered [ 2139.397420] Key type ._llcrypt unregistered [ 2149.025147] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2149.031083] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2151.636559] Key type ._llcrypt registered [ 2151.639350] Key type .llcrypt registered [ 2151.842355] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2151.875911] alg: No test for adler32 (adler32-zlib) [ 2152.921353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2153.103608] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2164.853342] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 02:11:06 (1787119866) [ 2164.997528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2165.047176] LNet: 104160:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2165.068633] LNetError: 104160:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2165.105443] LNet: Removed LNI 10.1.2.3@tcp [ 2165.194103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2166.209888] Key type .llcrypt unregistered [ 2166.214070] Key type ._llcrypt unregistered [ 2173.341389] Key type ._llcrypt registered [ 2173.345466] Key type .llcrypt registered [ 2173.595591] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2173.622742] alg: No test for adler32 (adler32-zlib) [ 2174.657957] LNet: Added LNI 192.168.201.49@tcp [8/256/0/180] [ 2174.722501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.49@tcp [ 2174.951620] LNet: 104460:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2174.960585] LNetError: 104460:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2176.043842] LNet: Removed LNI 192.168.201.49@tcp [ 2177.128479] Key type .llcrypt unregistered [ 2177.131210] Key type ._llcrypt unregistered [ 2190.528950] Key type ._llcrypt registered [ 2190.533294] Key type .llcrypt registered [ 2191.763367] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 02:11:33 (1787119893) [ 2192.769726] Key type .llcrypt unregistered [ 2192.773492] Key type ._llcrypt unregistered [ 2200.697245] Key type ._llcrypt registered [ 2200.699347] Key type .llcrypt registered [ 2200.889412] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2200.902536] alg: No test for adler32 (adler32-zlib) [ 2208.919684] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 02:11:50 (1787119910) [ 2209.298372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2209.350908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2209.441202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2209.479866] LNet: Added LNI 192.168.201.49@tcp [8/256/0/180] [ 2210.062452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2210.097624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 2210.124474] LNet: Removed LNI 192.168.201.49@tcp [ 2210.348157] LNet: 106218:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2210.713553] Key type .llcrypt unregistered [ 2210.716132] Key type ._llcrypt unregistered [ 2225.283733] Key type ._llcrypt registered [ 2225.287212] Key type .llcrypt registered [ 2227.096640] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 02:12:08 (1787119928) [ 2227.467838] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2227.477916] alg: No test for adler32 (adler32-zlib) [ 2228.481246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2228.544804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2228.568497] LNet: Added LNI 192.168.201.49@tcp [8/256/0/180] [ 2229.211377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.49@tcp [ 2229.242373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.201.49@tcp [ 2229.254783] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 2229.257507] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 2229.260294] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2229, 0, 0 [ 2229.264589] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 2229.267126] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 2229.270343] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 2229.273296] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2229, 0, 0 [ 2229.277442] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 2229.279905] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2229.282066] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 2229.284805] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2229, 0, 2279 [ 2229.288565] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 2229.290725] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2229.293767] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 2229.296695] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2229, 0, 2279 [ 2229.300834] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 2235.098230] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 02:12:17 (1787119937) [ 2235.567561] LNet: 107760:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2235.575413] LNetError: 107760:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2235.594270] LNet: Removed LNI 192.168.201.49@tcp [ 2236.217480] Key type .llcrypt unregistered [ 2236.219062] Key type ._llcrypt unregistered [ 2243.216992] Key type ._llcrypt registered [ 2243.218966] Key type .llcrypt registered [ 2243.542300] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2243.570365] alg: No test for adler32 (adler32-zlib) [ 2244.470408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2244.490585] LNet: Added LNI 192.168.201.49@tcp [8/256/0/180] [ 2252.344867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.149@tcp [ 2261.735729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2261.793232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2261.813413] LNet: Added LNI 192.168.201.49@tcp1 [8/256/0/180] [ 2262.857584] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2263.008574] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2263.016459] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2263.021066] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2265.509378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 2265.542827] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 2276.402246] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 02:12:58 (1787119978) [ 2276.547494] LNet: 109266:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2276.556645] LNetError: 109266:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2276.564202] LNet: Removed LNI 192.168.201.49@tcp [ 2277.599270] LNet: Removed LNI 192.168.201.49@tcp1 [ 2277.603761] LNetError: 109266:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2277.617504] LNetError: 109266:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2277.653642] LNet: Removed LNI 10.1.2.3@tcp99 [ 2278.547948] Key type .llcrypt unregistered [ 2278.549589] Key type ._llcrypt unregistered [ 2285.311240] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2285.317060] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2287.914504] Key type ._llcrypt registered [ 2287.916684] Key type .llcrypt registered [ 2288.146994] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2288.162505] alg: No test for adler32 (adler32-zlib) [ 2289.051784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2289.147678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2289.214727] LNet: Added LNI 192.168.201.49@tcp [8/256/0/180] [ 2290.052918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2290.073892] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2291.843876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 2291.890748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2291.929453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 2291.948550] LNetError: 110022:0:(peer.c:2362:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 2291.978803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2292.023360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.49@tcp [ 2292.482957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2292.590581] LNet: 110042:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2292.594975] LNetError: 110042:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2292.599727] LNet: Removed LNI 10.1.2.3@tcp [ 2293.696877] LNet: Removed LNI 192.168.201.49@tcp [ 2294.592519] Key type .llcrypt unregistered [ 2294.595359] Key type ._llcrypt unregistered [ 2307.804439] Key type ._llcrypt registered [ 2307.810203] Key type .llcrypt registered [ 2309.520495] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 02:13:30 (1787120010) [ 2310.303233] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2310.315038] alg: No test for adler32 (adler32-zlib) [ 2311.297922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2311.356703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2311.386905] LNet: Added LNI 192.168.201.49@tcp [8/256/0/180] [ 2312.396781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.49@tcp [ 2312.543838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg149-client@tcp [ 2320.953791] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 02:13:43 (1787120023) [ 2321.442860] LNet: 111582:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2321.469661] LNetError: 111582:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2321.492671] LNet: Removed LNI 192.168.201.49@tcp [ 2322.294227] Key type .llcrypt unregistered [ 2322.295657] Key type ._llcrypt unregistered [ 2329.519284] Key type ._llcrypt registered [ 2329.521823] Key type .llcrypt registered [ 2329.707515] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2329.741765] alg: No test for adler32 (adler32-zlib) [ 2330.713724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 2330.773161] LNet: Added LNI 192.168.201.49@tcp [8/256/0/180] [ 2330.823648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 2338.205056] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 02:14:00 (1787120040) [ 2340.003941] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 2341.658393] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 02:14:03 (1787120043) [ 2343.548133] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 2345.511275] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 02:14:07 (1787120047) [ 2345.677679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2345.731328] LNet: 112999:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2345.748710] LNetError: 112999:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2345.817172] LNet: Removed LNI 192.168.201.49@tcp [ 2345.900679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2345.969213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2345.990973] LNet: Added LNI 192.168.201.49@tcp [8/256/0/180] [ 2347.072609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2347.148738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2355.848793] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 02:14:17 (1787120057) [ 2356.599511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2356.635445] LNet: 113714:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2356.662030] LNetError: 113714:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2356.702382] LNet: Removed LNI 192.168.201.49@tcp [ 2356.742942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2356.831298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 2356.916216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2357.957443] Key type .llcrypt unregistered [ 2357.960062] Key type ._llcrypt unregistered [ 2370.699285] Key type ._llcrypt registered [ 2370.700729] Key type .llcrypt registered [ 2372.171090] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 02:14:33 (1787120073) [ 2372.724736] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2372.743135] alg: No test for adler32 (adler32-zlib) [ 2373.752727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2373.826885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2373.850697] LNet: Added LNI 192.168.201.49@tcp [8/256/0/180] [ 2374.601671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2374.642119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.49@tcp [ 2381.476046] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 02:14:43 (1787120083) [ 2381.597949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2381.633774] LNet: 115267:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2381.648126] LNetError: 115267:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2381.679259] LNet: Removed LNI 192.168.201.49@tcp [ 2381.700188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2381.779962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 2381.806245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 2389.231975] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 02:14:50 (1787120090) [ 2389.571544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2389.606633] LNet: 115853:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2389.722856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2389.824633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2390.500189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2390.550925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2390.605928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 2391.257507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2391.329872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2391.467283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2391.520586] LNet: 115887:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2391.559219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2391.622955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2392.001750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2392.083706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2392.142788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 2392.710649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2392.733400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2398.792885] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 02:15:00 (1787120100) [ 2399.203811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2399.255440] LNet: 116487:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2399.310401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2399.395581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2399.442407] LNet: Added LNI 192.168.201.49@tcp [8/256/0/180] [ 2400.254288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2400.287474] LNet: Added LNI 192.168.201.49@tcp2 [8/256/0/180] [ 2401.083567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2401.148407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.49@tcp [ 2401.514338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2401.556782] LNetError: 116767:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2401.568925] LNet: Removed LNI 192.168.201.49@tcp [ 2401.584841] LNet: Removed LNI 192.168.201.49@tcp2 [ 2401.628584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2401.710396] LNet: Added LNI 192.168.201.49@tcp [8/256/0/180] [ 2401.717167] LNet: Added LNI 192.168.201.49@tcp2 [8/256/0/180] [ 2401.727177] LNet: 116773:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2402.248871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.49@tcp [ 2402.429964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2403.487202] LNet: Removed LNI 192.168.201.49@tcp [ 2403.489118] LNetError: 116810:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2403.499872] LNetError: 116810:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 2403.522744] LNet: Removed LNI 192.168.201.49@tcp2 [ 2403.576542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2403.677601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2403.706204] LNet: Added LNI 192.168.201.49@tcp [8/256/0/180] [ 2404.512617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2404.539157] LNet: Added LNI 192.168.201.49@tcp2 [8/256/0/180] [ 2405.445517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.49@tcp [ 2405.484979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2405.554622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.49@tcp [ 2405.867387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2405.926753] LNet: 117094:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2405.941123] LNet: 117094:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 2405.948798] LNetError: 117094:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2405.958382] LNet: Removed LNI 192.168.201.49@tcp [ 2407.023381] LNet: Removed LNI 192.168.201.49@tcp2 [ 2407.085603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2407.226428] LNet: Added LNI 192.168.201.49@tcp [8/256/0/180] [ 2407.234498] LNet: Added LNI 192.168.201.49@tcp2 [8/256/0/180] [ 2407.246508] LNet: 117100:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2407.646225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.49@tcp [ 2407.888195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2407.922341] LNet: Removed LNI 192.168.201.49@tcp [ 2408.951441] LNet: Removed LNI 192.168.201.49@tcp2 [ 2408.982581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2409.079476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2409.118952] LNet: Added LNI 192.168.201.49@tcp [8/256/0/180] [ 2409.976615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2410.010477] LNet: Added LNI 192.168.201.49@tcp2 [8/256/0/180] [ 2410.917756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2410.964955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.49@tcp [ 2411.197344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2411.214496] LNetError: 117417:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2411.224624] LNetError: 117417:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 2411.230913] LNet: Removed LNI 192.168.201.49@tcp [ 2411.244730] LNet: Removed LNI 192.168.201.49@tcp2 [ 2411.264177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2411.317552] LNet: Added LNI 192.168.201.49@tcp [8/256/0/180] [ 2411.322616] LNet: Added LNI 192.168.201.49@tcp2 [8/256/0/180] [ 2411.327730] LNet: 117423:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2412.057258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.49@tcp [ 2412.291980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2413.343303] LNet: Removed LNI 192.168.201.49@tcp [ 2413.362673] LNet: Removed LNI 192.168.201.49@tcp2 [ 2413.406567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2413.525243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2413.546826] LNet: Added LNI 192.168.201.49@tcp [8/256/0/180] [ 2414.176722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2414.205637] LNet: Added LNI 192.168.201.49@tcp2 [8/256/0/180] [ 2415.140671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.49@tcp [ 2415.245099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2415.321420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.49@tcp [ 2415.576058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2415.621227] LNet: 117745:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2415.632606] LNet: 117745:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 2416.671148] LNet: Removed LNI 192.168.201.49@tcp [ 2417.706257] LNet: Removed LNI 192.168.201.49@tcp2 [ 2417.749491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2417.804218] LNet: Added LNI 192.168.201.49@tcp [8/256/0/180] [ 2417.820840] LNet: Added LNI 192.168.201.49@tcp2 [8/256/0/180] [ 2417.832549] LNet: 117751:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2418.525946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.49@tcp [ 2418.820529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2418.852238] LNet: Removed LNI 192.168.201.49@tcp [ 2419.878906] LNet: Removed LNI 192.168.201.49@tcp2 [ 2419.954824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2420.050686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2420.105340] LNet: Added LNI 192.168.201.49@tcp [8/256/0/180] [ 2420.998560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2421.022808] LNet: Added LNI 192.168.201.49@tcp2 [8/256/0/180] [ 2421.983738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2422.035719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2422.114313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.49@tcp [ 2422.518519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2422.548452] LNetError: 118084:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2422.561705] LNetError: 118084:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 2422.575488] LNet: Removed LNI 192.168.201.49@tcp [ 2422.594711] LNet: Removed LNI 192.168.201.49@tcp2 [ 2422.631954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2422.739479] LNet: Added LNI 192.168.201.49@tcp [8/256/0/180] [ 2422.751376] LNet: Added LNI 192.168.201.49@tcp2 [8/256/0/180] [ 2422.763256] LNet: 118090:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2423.201556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.49@tcp [ 2423.604549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2424.672819] LNet: Removed LNI 192.168.201.49@tcp [ 2424.686813] LNet: Removed LNI 192.168.201.49@tcp2 [ 2424.749166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2424.822973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2424.853900] LNet: Added LNI 192.168.201.49@tcp [8/256/0/180] [ 2426.171837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2426.216891] LNet: Added LNI 192.168.201.49@tcp2 [8/256/0/180] [ 2427.415914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.49@tcp [ 2427.475888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2427.508167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2427.569452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.49@tcp [ 2428.028297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2429.087921] LNet: Removed LNI 192.168.201.49@tcp [ 2430.119852] LNet: Removed LNI 192.168.201.49@tcp2 [ 2430.204072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2430.348392] LNet: Added LNI 192.168.201.49@tcp [8/256/0/180] [ 2430.357687] LNet: Added LNI 192.168.201.49@tcp2 [8/256/0/180] [ 2431.279745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.49@tcp [ 2438.675438] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 2440.636970] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 02:15:42 (1787120142) [ 2440.868979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2440.911798] LNet: 119210:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2440.918511] LNet: 119210:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 2440.930869] LNetError: 119210:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2440.942303] LNetError: 119210:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 2440.949169] LNet: Removed LNI 192.168.201.49@tcp [ 2442.025313] LNet: Removed LNI 192.168.201.49@tcp2 [ 2442.087911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2442.489646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2442.509966] LNet: Added LNI 192.168.201.49@tcp [8/256/0/180] [ 2443.336057] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2443.366729] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2444.398472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2444.417654] LNet: Removed LNI 10.1.2.3@tcp [ 2444.435585] LNet: Removed LNI 192.168.201.49@tcp [ 2444.454997] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2444.506913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 2444.531387] LNet: Added LNI 192.168.201.49@tcp [8/256/0/180] [ 2444.536182] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2445.054794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 2445.073665] LNet: Removed LNI 192.168.201.49@tcp [ 2445.083972] LNet: Removed LNI 10.1.2.3@tcp [ 2452.221142] Lustre: DEBUG MARKER: == sanity-lnet test 475: per-operation latency stats populate and reset ========================================================== 02:15:53 (1787120153) [ 2452.305771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2452.361833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2452.461315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2452.493513] LNet: Added LNI 192.168.201.49@tcp [8/256/0/180] [ 2453.159813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2453.214935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.49@tcp [ 2453.268280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.49@tcp [ 2453.295519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.49@tcp [ 2453.321825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.49@tcp [ 2453.349862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.49@tcp [ 2453.373709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.49@tcp [ 2453.404579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.49@tcp [ 2453.430741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.49@tcp [ 2453.477693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.49@tcp [ 2453.515754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.49@tcp [ 2453.633050] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2460.740381] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 02:16:02 (1787120162) [ 2461.212424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2462.310764] LNet: Removed LNI 192.168.201.49@tcp [ 2462.397515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2462.507576] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2462.582464] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2462.602534] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2462.611515] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2465.134616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2465.188107] LNet: Added LNI 192.168.201.49@tcp [8/256/0/180] [ 2466.084190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2466.110264] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2468.540320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.49@tcp [ 2468.595650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 2468.674162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.201.49@tcp -m PUT -e local_timeout -r 1 [ 2468.742878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 2468.809717] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2468.813990] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2469.114141] LNetError: 120893:0:(lib-move.c:4801:lnet_parse()) 192.168.201.49@tcp, src 192.168.201.49@tcp: Dropping PUT (error -108 looking up sender) [ 2470.178179] LNet: Removed LNI 10.1.2.3@tcp [ 2471.207237] LNet: Removed LNI 192.168.201.49@tcp [ 2471.662766] Key type .llcrypt unregistered [ 2471.668380] Key type ._llcrypt unregistered [ 2484.650851] Key type ._llcrypt registered [ 2484.659353] Key type .llcrypt registered [ 2485.926572] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 02:16:27 (1787120187) [ 2486.381183] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2486.405647] alg: No test for adler32 (adler32-zlib) [ 2487.350683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2487.820620] LNet: 122218:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2488.544966] Key type .llcrypt unregistered [ 2488.548474] Key type ._llcrypt unregistered [ 2496.322657] Key type ._llcrypt registered [ 2496.329326] Key type .llcrypt registered [ 2496.548791] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2496.563023] alg: No test for adler32 (adler32-zlib) [ 2497.457317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2497.473586] LNet: Added LNI 192.168.201.49@tcp [8/256/0/180] [ 2506.761314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.149@tcp [ 2507.026859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 2507.714576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.201.149@tcp [ 2507.745346] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.149@tcp [ 2507.956507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2507.985411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2508.010461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2508.041799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2508.067614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2508.090856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2508.123027] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2508.175869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2508.213810] LNet: 122672:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2508.229508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2508.277993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2508.332800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2508.375692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2508.398844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2508.430558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2509.124640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2509.441876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2509.521380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2509.559733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2509.592080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2509.625926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2509.718239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2509.779049] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2509.855402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2509.886382] LNet: 122789:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2509.909290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2509.958750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2509.993996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2510.039357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2510.074887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2510.127319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2510.782521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.201.149@tcp [ 2511.050817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2511.088683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2511.133570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2511.199704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2511.240645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2511.278264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2511.337400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2511.371771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2511.394184] LNet: 122907:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2511.424302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2511.465432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2511.503037] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2511.531619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2511.591384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2511.654907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2512.295608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2512.605198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2512.635956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2512.673609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2512.718541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2512.762145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2512.798368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2512.829597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2512.863367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2512.905116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2512.957306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2513.009374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2513.053268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2513.083307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2513.112346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2513.514236] LNet: 123074:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2513.525627] LNetError: 123074:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2513.537928] LNet: Removed LNI 192.168.201.49@tcp [ 2513.965886] Key type .llcrypt unregistered [ 2513.967587] Key type ._llcrypt unregistered [ 2534.886924] Key type ._llcrypt registered [ 2534.889712] Key type .llcrypt registered [ 2536.202505] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 02:17:17 (1787120237) [ 2536.680365] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2536.693593] alg: No test for adler32 (adler32-zlib) [ 2537.658318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2538.178589] LNet: 123962:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2538.923523] Key type .llcrypt unregistered [ 2538.930791] Key type ._llcrypt unregistered [ 2546.204470] Key type ._llcrypt registered [ 2546.209211] Key type .llcrypt registered [ 2546.412896] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2546.427433] alg: No test for adler32 (adler32-zlib) [ 2547.404807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2547.473597] LNet: Added LNI 192.168.201.49@tcp [8/256/0/180] [ 2557.430824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.149@tcp [ 2568.359491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2568.424318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2568.469669] LNet: Added LNI 192.168.201.49@tcp1 [8/256/0/180] [ 2569.391578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 2569.911858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.201.149@tcp [ 2570.020175] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.149@tcp [ 2570.148456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2570.210151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2570.267510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2570.331734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2570.361481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2570.402770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2570.442723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2570.490913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2570.521740] LNet: 124618:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2570.556748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2570.596101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2570.642811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2570.669120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2570.704552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2570.758245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2571.343044] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2571.537978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2571.552471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2571.598854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2571.634527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2571.677994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2571.713980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2571.736622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2571.755331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2571.775823] LNet: 124736:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2571.795515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2571.836470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2571.869293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2571.905628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2571.940818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2571.974534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2572.526969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.201.149@tcp [ 2572.731765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2572.752831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2572.785098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2572.835672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2572.867713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2572.896436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2572.928602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2572.960663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2572.978946] LNet: 124853:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2572.995699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2573.024696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2573.047680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2573.074835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2573.102595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2573.141888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2573.694414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2573.906704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2573.925613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2573.942847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2573.975920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2574.004741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2574.026782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2574.057539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2574.084601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2574.122624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2574.155629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2574.187328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2574.220994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2574.258472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2574.295271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2578.357612] LNet: 125055:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2578.371488] LNetError: 125055:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2578.383719] LNet: Removed LNI 192.168.201.49@tcp [ 2578.414282] LNet: Removed LNI 192.168.201.49@tcp1 [ 2579.046452] Key type .llcrypt unregistered [ 2579.050692] Key type ._llcrypt unregistered [ 2600.457878] Key type ._llcrypt registered [ 2600.459900] Key type .llcrypt registered [ 2601.682122] Lustre: DEBUG MARKER: == sanity-lnet test 525: keep reachable peer on local net delete ========================================================== 02:18:23 (1787120303) [ 2602.179256] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2602.208790] alg: No test for adler32 (adler32-zlib) [ 2603.127399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2603.202160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2603.229068] LNet: Added LNI 192.168.201.49@tcp [8/256/0/180] [ 2603.940739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2603.960329] LNet: Added LNI 192.168.201.49@tcp2 [8/256/0/180] [ 2604.946702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.49@tcp [ 2605.062930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp [ 2606.111167] LNet: Removed LNI 192.168.201.49@tcp [ 2612.691127] Lustre: DEBUG MARKER: == sanity-lnet test 550: DD-off: keep route up when gw reachable via another local NI ========================================================== 02:18:34 (1787120314) [ 2613.952593] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_550 Need at least 2 local interfaces for multi-rail [ 2615.603900] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 02:18:37 (1787120317) [ 2623.367289] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 02:18:45 (1787120325) [ 2631.475604] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 02:18:53 (1787120333) [ 2641.280737] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 02:19:02 (1787120342) [ 2648.906478] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 02:19:10 (1787120350) [ 2657.743841] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 02:19:19 (1787120359) [ 2666.224658] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 02:19:27 (1787120367) [ 2673.951686] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 02:19:36 (1787120376) [ 2681.361799] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 02:19:43 (1787120383) [ 2688.022874] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 02:19:50 (1787120390) [ 2695.289787] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 02:19:57 (1787120397) [ 2702.249499] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 02:20:04 (1787120404) [ 2709.311405] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 02:20:11 (1787120411) [ 2717.067290] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 02:20:19 (1787120419) [ 2725.090399] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 02:20:27 (1787120427) [ 2732.520864] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 02:20:34 (1787120434) [ 2740.654910] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 02:20:42 (1787120442) [ 2749.816442] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 02:20:51 (1787120451) [ 2749.898614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2749.929745] LNet: 136752:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2749.936461] LNetError: 136752:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2749.956481] LNet: Removed LNI 192.168.201.49@tcp2 [ 2750.031625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2750.221341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2750.255297] LNet: Added LNI 192.168.201.49@tcp [8/256/0/180] [ 2751.510433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2751.546137] LNet: 136900:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2751.558603] LNetError: 136900:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2751.584156] LNet: Removed LNI 192.168.201.49@tcp [ 2751.645492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2751.693950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2751.712546] LNet: Added LNI 192.168.201.49@tcp [8/256/0/180] [ 2758.741459] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 02:21:00 (1787120460) [ 2767.754297] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 02:21:09 (1787120469) [ 2775.833831] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 02:21:17 (1787120477) [ 2784.289811] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 02:21:25 (1787120485) [ 2791.255825] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 02:21:33 (1787120493) [ 2797.967821] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 02:21:40 (1787120500) [ 2804.638065] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 02:21:46 (1787120506) [ 2812.245888] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 02:21:54 (1787120514) [ 2819.392562] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 02:22:01 (1787120521) [ 2826.806383] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 02:22:08 (1787120528) [ 2833.296181] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 02:22:15 (1787120535) [ 2841.498324] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 02:22:23 (1787120543) [ 2848.567496] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 2549 sec ========== 02:22:30 (1787120550) [ 2849.251305] LNet: 144365:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2849.262210] LNetError: 144365:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2849.274280] LNet: Removed LNI 192.168.201.49@tcp [ 2849.768674] Key type .llcrypt unregistered [ 2849.776480] Key type ._llcrypt unregistered [ 2863.587698] Key type ._llcrypt registered [ 2863.589736] Key type .llcrypt registered [ 2863.826841] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2863.858537] alg: No test for adler32 (adler32-zlib) [ 2865.239391] Lustre: Lustre: Build Version: 2.17.57_1_g9da61ce [ 2865.642761] LNet: Added LNI 192.168.201.49@tcp [8/256/0/180] [ 2867.399204] Key type lgssc registered [ 2868.919519] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2953.420214] Lustre: Mounted lustre-client - version 2.17.57_1_g9da61ce [ 2961.815355] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2971.350860] Lustre: Unmounted lustre-client [ 3001.583184] Key type lgssc unregistered [ 3001.946781] LNet: 147327:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3001.955614] LNetError: 147327:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3001.978380] LNet: Removed LNI 192.168.201.49@tcp [ 3002.841418] Key type .llcrypt unregistered [ 3002.843828] Key type ._llcrypt unregistered