[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 455826013 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001010] APIC: Switch to symmetric I/O mode setup [ 0.002334] x2apic enabled [ 0.003006] Switched APIC routing to physical x2apic. [ 0.004013] kvm-guest: setup PV IPIs [ 0.006684] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007020] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008011] pid_max: default: 32768 minimum: 301 [ 0.009140] LSM: Security Framework initializing [ 0.010045] Yama: becoming mindful. [ 0.011035] SELinux: Initializing. [ 0.012033] *** VALIDATE selinux *** [ 0.020145] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.024841] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.026102] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027120] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028143] *** VALIDATE tmpfs *** [ 0.029484] *** VALIDATE proc *** [ 0.031227] *** VALIDATE cgroup *** [ 0.032011] *** VALIDATE cgroup2 *** [ 0.034089] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.035179] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.036009] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.037030] Spectre V2 : User space: Vulnerable [ 0.038009] Speculative Store Bypass: Vulnerable [ 0.041574] debug: unmapping init [mem 0xffffffffb5459000-0xffffffffb5460fff] [ 0.044111] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045791] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046028] ... version: 2 [ 0.047015] ... bit width: 48 [ 0.048013] ... generic registers: 4 [ 0.049012] ... value mask: 0000ffffffffffff [ 0.050012] ... max period: 00007fffffffffff [ 0.051017] ... fixed-purpose events: 3 [ 0.052009] ... event mask: 000000070000000f [ 0.053305] rcu: Hierarchical SRCU implementation. [ 0.055497] smp: Bringing up secondary CPUs ... [ 0.056655] x86: Booting SMP configuration: [ 0.057037] .... node #0, CPUs: #1 #2 #3 [ 0.060616] smp: Brought up 1 node, 4 CPUs [ 0.062019] smpboot: Max logical packages: 1 [ 0.063015] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.145044] node 0 deferred pages initialised in 80ms [ 0.148100] devtmpfs: initialized [ 0.149211] x86/mm: Memory block size: 128MB [ 0.151670] gcov: version magic: 0x41383552 [ 0.152697] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.155114] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.157354] pinctrl core: initialized pinctrl subsystem [ 0.159184] [ 0.159825] ************************************************************* [ 0.162013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.164013] ** ** [ 0.166010] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.168013] ** ** [ 0.170010] ** This means that this kernel is built to expose internal ** [ 0.171008] ** IOMMU data structures, which may compromise security on ** [ 0.173007] ** your system. ** [ 0.174009] ** ** [ 0.176014] ** If you see this message and you are not debugging the ** [ 0.178009] ** kernel, report this immediately to your vendor! ** [ 0.179009] ** ** [ 0.181011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.182009] ************************************************************* [ 0.184760] NET: Registered protocol family 16 [ 0.186359] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.188041] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.190056] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.193173] cpuidle: using governor menu [ 0.194621] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.195455] PCI: Using configuration type 1 for base access [ 0.196133] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.202100] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.204032] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.207061] cryptd: max_cpu_qlen set to 1000 [ 0.210378] ACPI: Added _OSI(Module Device) [ 0.212013] ACPI: Added _OSI(Processor Device) [ 0.213010] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.214009] ACPI: Added _OSI(Processor Aggregator Device) [ 0.217718] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.223424] ACPI: Interpreter enabled [ 0.224117] ACPI: PM: (supports S0 S3 S4 S5) [ 0.226012] ACPI: Using IOAPIC for interrupt routing [ 0.228128] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.229401] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.239826] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.242040] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.244019] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.246109] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.249023] acpiphp: Slot [2] registered [ 0.249952] acpiphp: Slot [5] registered [ 0.251071] acpiphp: Slot [6] registered [ 0.252015] acpiphp: Slot [3] registered [ 0.252954] acpiphp: Slot [4] registered [ 0.254062] acpiphp: Slot [7] registered [ 0.254995] acpiphp: Slot [8] registered [ 0.255054] acpiphp: Slot [9] registered [ 0.255988] acpiphp: Slot [10] registered [ 0.257093] acpiphp: Slot [11] registered [ 0.258096] acpiphp: Slot [12] registered [ 0.258999] acpiphp: Slot [13] registered [ 0.260059] acpiphp: Slot [14] registered [ 0.261031] acpiphp: Slot [15] registered [ 0.261923] acpiphp: Slot [16] registered [ 0.263134] acpiphp: Slot [17] registered [ 0.265108] acpiphp: Slot [18] registered [ 0.266113] acpiphp: Slot [19] registered [ 0.268111] acpiphp: Slot [20] registered [ 0.270111] acpiphp: Slot [21] registered [ 0.272196] acpiphp: Slot [22] registered [ 0.273115] acpiphp: Slot [23] registered [ 0.275124] acpiphp: Slot [24] registered [ 0.276092] acpiphp: Slot [25] registered [ 0.277102] acpiphp: Slot [26] registered [ 0.279088] acpiphp: Slot [27] registered [ 0.280062] acpiphp: Slot [28] registered [ 0.280995] acpiphp: Slot [29] registered [ 0.282081] acpiphp: Slot [30] registered [ 0.283111] acpiphp: Slot [31] registered [ 0.284096] PCI host bridge to bus 0000:00 [ 0.285016] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.287020] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.289015] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.290019] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.292015] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.294016] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.295168] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.298145] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.301273] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.309021] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.313060] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.318031] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.320252] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.323030] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.325559] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.327886] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.331044] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.333846] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.338015] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.348017] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.353018] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.359150] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.374034] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.381028] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.396026] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.410549] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.422021] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.428019] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.453021] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.466909] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.469523] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.472950] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.475459] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.478306] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.484052] iommu: Default domain type: Passthrough [ 0.486518] SCSI subsystem initialized [ 0.488143] ACPI: bus type USB registered [ 0.489181] usbcore: registered new interface driver usbfs [ 0.492104] usbcore: registered new interface driver hub [ 0.493086] usbcore: registered new device driver usb [ 0.495181] pps_core: LinuxPPS API ver. 1 registered [ 0.497015] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.501083] PTP clock support registered [ 0.504144] EDAC MC: Ver: 3.0.0 [ 0.505278] PCI: Using ACPI for IRQ routing [ 0.507085] NetLabel: Initializing [ 0.508015] NetLabel: domain hash size = 128 [ 0.510018] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.511194] NetLabel: unlabeled traffic allowed by default [ 0.514198] vgaarb: loaded [ 0.516357] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.518024] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.527152] clocksource: Switched to clocksource kvm-clock [ 0.641763] VFS: Disk quotas dquot_6.6.0 [ 0.643582] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.646407] *** VALIDATE ramfs *** [ 0.647815] *** VALIDATE hugetlbfs *** [ 0.649617] pnp: PnP ACPI init [ 0.652285] pnp: PnP ACPI: found 6 devices [ 0.671378] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.674666] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.676996] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.679347] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.681875] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.684361] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.687229] NET: Registered protocol family 2 [ 0.689792] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.694820] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.698636] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.704273] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.707945] TCP: Hash tables configured (established 65536 bind 65536) [ 0.711124] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.714351] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.717264] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.720364] NET: Registered protocol family 1 [ 0.722946] RPC: Registered named UNIX socket transport module. [ 0.725420] RPC: Registered udp transport module. [ 0.726901] RPC: Registered tcp transport module. [ 0.728468] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.730615] NET: Registered protocol family 44 [ 0.732382] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.735091] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.737632] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.740291] PCI: CLS 0 bytes, default 64 [ 0.742388] Unpacking initramfs... [ 2.116495] debug: unmapping init [mem 0xffff91bc3cc64000-0xffff91bc3ffcffff] [ 2.120532] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.122871] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.126132] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.622486] Initialise system trusted keyrings [ 2.624283] Key type blacklist registered [ 2.626379] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.635582] zbud: loaded [ 2.638675] *** VALIDATE nfs *** [ 2.639876] *** VALIDATE nfs4 *** [ 2.641552] pstore: using deflate compression [ 2.644722] Platform Keyring initialized [ 2.749809] NET: Registered protocol family 38 [ 2.752157] Key type asymmetric registered [ 2.753961] Asymmetric key parser 'x509' registered [ 2.756627] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.761175] io scheduler mq-deadline registered [ 2.763594] io scheduler kyber registered [ 2.765813] io scheduler bfq registered [ 2.768592] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.772837] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.777045] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.782712] ACPI: Power Button [PWRF] [ 2.789280] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.795720] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.811543] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.839065] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.866483] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.871768] Non-volatile memory driver v1.3 [ 2.873572] Linux agpgart interface v0.103 [ 2.906314] virtio_blk virtio1: [vda] 145936 512-byte logical blocks (74.7 MB/71.3 MiB) [ 2.909314] vda: detected capacity change from 0 to 74719232 [ 2.927995] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 2.932175] vdb: detected capacity change from 0 to 1073741824 [ 2.946066] libphy: Fixed MDIO Bus: probed [ 2.954319] usbcore: registered new interface driver usbserial_generic [ 2.957898] usbserial: USB Serial support registered for generic [ 2.961452] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 2.966297] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 2.968361] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 2.970988] mousedev: PS/2 mouse device common for all mice [ 2.973487] rtc_cmos 00:05: RTC can wake from S4 [ 2.976361] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 2.976477] rtc_cmos 00:05: registered as rtc0 [ 2.981328] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 2.984966] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 2.985218] intel_pstate: CPU model not supported [ 2.990411] hid: raw HID events driver (C) Jiri Kosina [ 2.992734] usbcore: registered new interface driver usbhid [ 2.994440] usbhid: USB HID core driver [ 2.996186] drop_monitor: Initializing network drop monitor service [ 2.996545] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 2.998357] Initializing XFRM netlink socket [ 3.002070] NET: Registered protocol family 10 [ 3.004615] Segment Routing with IPv6 [ 3.006383] NET: Registered protocol family 17 [ 3.009795] mpls_gso: MPLS GSO support [ 3.015838] RAS: Correctable Errors collector initialized. [ 3.018049] AVX version of gcm_enc/dec engaged. [ 3.019740] AES CTR mode by8 optimization enabled [ 3.091529] sched_clock: Marking stable (3091507271, 0)->(3918163806, -826656535) [ 3.094464] registered taskstats version 1 [ 3.096299] Loading compiled-in X.509 certificates [ 3.098567] zswap: loaded using pool lzo/zbud [ 3.123132] Key type big_key registered [ 3.134828] Key type encrypted registered [ 3.136135] ima: No TPM chip found, activating TPM-bypass! [ 3.138102] ima: Allocated hash algorithm: sha1 [ 3.139610] ima: No architecture policies found [ 3.141389] evm: Initialising EVM extended attributes: [ 3.142796] evm: security.selinux [ 3.143590] evm: security.ima [ 3.144396] evm: security.capability [ 3.145587] evm: HMAC attrs: 0x1 [ 3.147581] rtc_cmos 00:05: setting system clock to 2026-08-24 18:54:22 UTC (1787597662) [ 3.153371] debug: unmapping init [mem 0xffffffffb6403000-0xffffffffb65fffff] [ 3.155960] debug: unmapping init [mem 0xffffffffb5182000-0xffffffffb5458fff] [ 3.166102] Write protecting the kernel read-only data: 28672k [ 3.168945] debug: unmapping init [mem 0xffffffffb3803000-0xffffffffb39fffff] [ 3.171523] debug: unmapping init [mem 0xffffffffb4114000-0xffffffffb41fffff] [ 3.200045] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.207751] systemd[1]: Detected virtualization kvm. [ 3.209462] systemd[1]: Detected architecture x86-64. [ 3.211075] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.237317] systemd[1]: No hostname configured. [ 3.239059] systemd[1]: Set hostname to . [ 3.241214] random: systemd: uninitialized urandom read (16 bytes read) [ 3.243654] systemd[1]: Initializing machine ID from random generator. [ 3.288612] random: ln: uninitialized urandom read (6 bytes read) [ 3.379190] random: systemd: uninitialized urandom read (16 bytes read) [ 3.381833] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 3.387450] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 3.391236] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ OK ] Listening on udev Control Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Paths. [ OK ] Reached target Slices. [ OK ] Reached target Swap. [ OK ] Listening on Journal Socket. Starting Create Volatile Files and Directories... Starting Create list of required st…ce nodes for the current kernel... Starting Setup Virtual Console... [ OK ] Reached target Sockets. Starting Apply Kernel Variables... Starting Journal Service... [ OK ] Started Memstrack Anylazing Service. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 3.936632] device-mapper: uevent: version 1.0.3 [ 3.938191] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 4.578101] virtio_net virtio0 ens2: renamed from eth0 [ 4.618758] scsi host0: ata_piix [ 4.710249] scsi host1: ata_piix [ 4.711815] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 4.714263] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 8.923193] dracut-initqueue[587]: RTNETLINK answers: File exists [ 9.649607] random: crng init done [ 9.650916] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 9.909610] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Swap. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Apply Kernel Variables. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 10.956879] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.195093] SELinux: Disabled at runtime. [ 11.254294] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.262445] systemd[1]: Detected virtualization kvm. [ 11.264416] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 11.719174] systemd[1]: initrd-switch-root.service: Succeeded. [ 11.722448] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 11.727203] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 11.731270] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 11.734682] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 11.741146] systemd[1]: Starting Journal Service... Starting Journal Service... [ 11.744381] systemd[1]: proc-sys-fs-binfmt_misc.automount: Refusing to start, unit to trigger not loaded. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Listening on initctl Compatibility Named Pipe. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice system-getty.slice. [ OK ] Created slice system-serial\x2dgetty.slice. Starting Remount Root and Kernel File Systems... [ OK ] Reached target rpc_pipefs.target. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. Mounting Huge Pages File System... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Created slice User and Session Slice. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Listening on Process Core Dump Socket. [ OK ] Stopped target Initrd Root File System. [ OK ] Created slice system-sshd\x2dkeygen.slice. Mounting Kernel Debug File System... Mounting POSIX Message Queue File System... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Slices. Activating swap /dev/disk/by-label/SWAP... Starting Apply Kernel Variables... [ 11.894550] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Apply Kernel Variables. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Mounted /mnt. [ OK ] Started udev Coldplug all Devices. [ 12.127559] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 12.934492] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.011970] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.441803] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.487638] EDAC sbridge: Ver: 1.1.2 [ 16.618792] Key type dns_resolver registered [ 17.500505] NFS: Registering the id_resolver key type [ 17.503800] Key type id_resolver registered [ 17.505641] Key type id_legacy registered [* ] A start job is running for Configur…-only root support (6s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Mark the need to relabel after reboot. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Restore /run/initramfs on shutdown... Starting Login Service... [ OK ] Started irqbalance daemon. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Login Service. [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg209-client login: [ 90.307247] libcfs: loading out-of-tree module taints kernel. [ 90.379421] Key type ._llcrypt registered [ 90.409448] Key type .llcrypt registered [ 91.020629] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 91.031815] alg: No test for adler32 (adler32-zlib) [ 92.693501] Lustre: Lustre: Build Version: 2.17.57_1_g9ee77b6 [ 93.639948] LNet: Added LNI 192.168.202.9@tcp [8/256/0/180] [ 95.551184] Key type lgssc registered [ 97.736860] Lustre: Echo OBD driver; http://www.lustre.org/ [ 240.160853] hrtimer: interrupt took 6664611 ns [ 241.116792] Lustre: Mounted lustre-client - version 2.17.57_1_g9ee77b6 [ 246.142514] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 263.135126] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing check_logdir /tmp/testlogs/ [ 266.720526] Lustre: lustre-OST0000-osc-ffff91bc82f83000: disconnect after 24s idle [ 270.418307] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing yml_node [ 274.812636] Lustre: DEBUG MARKER: Client: 2.17.57.1 [ 278.035571] Lustre: DEBUG MARKER: MDS: 2.17.57.1 [ 281.271766] Lustre: DEBUG MARKER: OSS: 2.17.57.1 [ 282.982615] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Mon Aug 24 14:59:01 EDT 2026 [ 301.139690] Lustre: Unmounted lustre-client [ 328.523487] Key type lgssc unregistered [ 328.812138] LNet: 8435:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 328.825303] LNetError: 8435:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 328.856612] LNet: Removed LNI 192.168.202.9@tcp [ 329.675231] Key type .llcrypt unregistered [ 329.677392] Key type ._llcrypt unregistered [ 358.333820] Key type ._llcrypt registered [ 358.345520] Key type .llcrypt registered [ 358.567314] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 358.578262] alg: No test for adler32 (adler32-zlib) [ 359.474595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 359.551068] LNet: Added LNI 192.168.202.9@tcp [8/256/0/180] [ 359.572935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 364.128691] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 364.405155] LNet: 9324:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 364.417939] LNetError: 9324:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 364.434991] LNet: Removed LNI 192.168.202.9@tcp [ 365.185462] Key type .llcrypt unregistered [ 365.190122] Key type ._llcrypt unregistered [ 372.228615] Key type ._llcrypt registered [ 372.230521] Key type .llcrypt registered [ 373.885183] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 15:00:31 (1787598031) [ 374.246601] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 374.256938] alg: No test for adler32 (adler32-zlib) [ 375.102405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 375.171577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 375.200474] LNet: 9742:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 375.217564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 375.261866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 375.286231] LNet: 9749:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 383.169231] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 15:00:41 (1787598041) [ 383.302706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 383.364135] LNet: 10321:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 383.386890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 383.559879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 390.812739] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 15:00:48 (1787598048) [ 391.057640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 391.095797] LNet: 10914:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 391.116559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 391.205956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 398.249210] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 15:00:56 (1787598056) [ 398.496465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 398.544828] LNet: 11498:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 398.562824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 398.612128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 405.130536] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 15:01:03 (1787598063) [ 405.448726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 405.485599] LNet: 12097:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 405.515533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 405.635252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 406.184977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 412.538798] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 15:01:10 (1787598070) [ 412.913694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 412.941985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 413.005075] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 413.631625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 414.087569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 420.667318] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 15:01:18 (1787598078) [ 421.023543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 421.043818] LNet: 13284:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 421.048295] LNet: 13284:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 421.069890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 421.147192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 428.520373] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 15:01:26 (1787598086) [ 428.694973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 428.781058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 428.904698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 428.945738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 429.675663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 429.728937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 430.430967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 430.492224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 431.055846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 431.135136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 431.585750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 431.632061] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 432.140253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 432.186666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 432.662396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 432.696884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 432.972725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 432.997737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 433.455330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 433.487468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 433.830466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 433.860256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 434.346694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 434.395107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 434.815719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 434.847149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 435.349576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 435.409264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 442.416552] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 15:01:40 (1787598100) [ 442.679618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 442.722416] LNet: 14551:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 442.728258] LNet: 14551:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 442.748840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 442.855780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 442.891641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 450.077520] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 15:01:47 (1787598107) [ 450.176855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 450.222745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 450.297990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 450.325485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 457.105598] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 15:01:54 (1787598114) [ 457.240631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 457.319552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 457.473931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 457.572813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 465.227414] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 15:02:03 (1787598123) [ 465.325934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 465.388592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 465.468835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 465.525651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 473.195770] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 15:02:11 (1787598131) [ 473.489748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 473.523927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 473.586930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 473.620576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 481.272305] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 15:02:19 (1787598139) [ 481.554186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 481.589584] LNet: 17483:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 481.593875] LNet: 17483:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 481.608551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 481.700662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 481.746566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 489.148607] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 15:02:26 (1787598146) [ 489.290333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 489.366617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 489.493683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 490.124880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 498.017879] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 15:02:35 (1787598155) [ 498.151967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 498.208453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 498.531152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 498.861849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 506.620747] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 15:02:44 (1787598164) [ 507.077566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 507.128032] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 507.266623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 507.612655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 514.610729] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 15:02:52 (1787598172) [ 514.835439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 514.866766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 515.306358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 515.645510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 521.989289] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 15:03:00 (1787598180) [ 522.193842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 522.233817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 522.296880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 522.360777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 529.315361] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 15:03:06 (1787598186) [ 529.401778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 529.428301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 529.496832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 529.597219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 537.161745] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 15:03:15 (1787598195) [ 537.519802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 537.572506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 537.686594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 544.434407] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 15:03:22 (1787598202) [ 544.599807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 544.681506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 544.786817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 552.107185] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 15:03:30 (1787598210) [ 552.235896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 552.265903] LNet: 22915:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 552.273797] LNet: 22915:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 8 previous similar messages [ 552.310806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 552.487568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 552.997870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 560.988798] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 15:03:38 (1787598218) [ 561.182826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 561.258657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 561.347207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 561.380823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 569.192806] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 15:03:47 (1787598227) [ 569.347685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 569.411685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 569.538662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 569.582620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 576.536627] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 15:03:54 (1787598234) [ 576.633250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 576.687773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 576.827457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 576.871189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 584.370717] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 15:04:01 (1787598241) [ 584.508428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 584.562994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 584.723407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 584.777685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 592.810692] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 15:04:10 (1787598250) [ 592.905573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 592.953549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 593.045685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 593.130658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 593.154579] LNetError: 25872:0:(peer.c:2362:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 593.197868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 593.233730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 599.085343] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 15:04:17 (1787598257) [ 599.352396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 599.432401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 599.535374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.CdusR4JbCs [ 599.654809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.CdusR4JbCs [ 599.725717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.CdusR4JbCs [ 605.958958] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 15:04:23 (1787598263) [ 606.441671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 606.481716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 606.540984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 606.572484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 606.642635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 613.280764] Lustre: DEBUG MARKER: == sanity-lnet test 40: LNetAddPeer merges a pre-existing non-primary peer NID ========================================================== 15:04:31 (1787598271) [ 613.551459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 613.601558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 613.758049] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 613.802321] LNet: Added LNI 192.168.202.9@tcp [8/256/0/180] [ 615.572510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.9@tcp [ 616.926531] Lustre: Lustre: Build Version: 2.17.57_1_g9ee77b6 [ 618.896076] Key type lgssc registered [ 620.122891] Lustre: Echo OBD driver; http://www.lustre.org/ [ 623.439632] LNetError: 27634:0:(lib-move.c:2158:lnet_handle_find_routed_path()) peer 192.168.202.9@tcp99 has no available nets [ 626.046866] Key type lgssc unregistered [ 626.349930] LNetError: 28745:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 626.376283] LNet: Removed LNI 192.168.202.9@tcp [ 627.047728] Key type .llcrypt unregistered [ 627.053934] Key type ._llcrypt unregistered [ 633.981537] Key type ._llcrypt registered [ 633.984264] Key type .llcrypt registered [ 635.321906] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 15:04:53 (1787598293) [ 635.791843] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 635.798512] alg: No test for adler32 (adler32-zlib) [ 636.699593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 636.784733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 639.044923] LNet: Message forwarding will be enabled in 2s [ 642.143121] LNet: Message forwarding enabled [ 642.490779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 642.512734] LNet: Message forwarding will stop in 2s to 6s [ 647.343821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 648.580807] LNet: Message forwarding disabled [ 651.533504] LNet: Message forwarding will be enabled in 2s [ 654.751199] LNet: Message forwarding enabled [ 660.457584] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 15:05:18 (1787598318) [ 660.528888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 660.556689] LNet: 30149:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 662.183761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 662.345749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 669.443381] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 15:05:27 (1787598327) [ 669.506767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 669.543812] LNet: 30730:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 669.590927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 669.686773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 669.729689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 669.760972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 669.781825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 669.813629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 669.837767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 669.860996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 669.894644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 669.927790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 669.950612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 669.996897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 670.033361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 670.065243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 670.108733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 670.146877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 670.174362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 670.208929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 670.246349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 670.285579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 670.305188] LNetError: 30781:0:(router.c:625:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 678.154681] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 15:05:36 (1787598336) [ 678.342835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 678.376931] LNet: 31353:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 678.394302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 678.458439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 685.313625] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 15:05:43 (1787598343) [ 687.832906] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg209-server [ 689.897961] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 15:05:47 (1787598347) [ 689.979662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 690.019906] LNet: 32276:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 690.043253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 690.119963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 690.156769] LNet: Added LNI 192.168.202.9@tcp [8/256/0/180] [ 691.172086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.202.[10-14]@tcp [ 691.211612] LNet: 32417:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 691.258556] LNetError: 32295:0:(lib-lnet.h:1279:lnet_set_route_aliveness()) route to tcp1 through 192.168.202.12@tcp has gone from up to down [ 691.317985] LNetError: 32285:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.202.11@tcp is being used as a gateway but routing feature is not turned on [ 692.267536] LNetError: 32285:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.202.11@tcp is being used as a gateway but routing feature is not turned on [ 692.294748] LNetError: 32285:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 5 previous similar messages [ 693.311861] LNetError: 32285:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.202.11@tcp is being used as a gateway but routing feature is not turned on [ 693.326837] LNetError: 32285:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 2 previous similar messages [ 695.332899] LNetError: 32285:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.202.13@tcp is being used as a gateway but routing feature is not turned on [ 695.347670] LNetError: 32285:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 5 previous similar messages [ 698.791550] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 15:05:56 (1787598356) [ 699.475943] LNetError: 32285:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.202.10@tcp is being used as a gateway but routing feature is not turned on [ 699.491689] LNetError: 32285:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 13 previous similar messages [ 701.096424] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg209-server [ 703.390568] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 15:06:00 (1787598360) [ 703.484811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 703.510074] LNet: 33347:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 703.525556] LNetError: 33347:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 703.544896] LNet: Removed LNI 192.168.202.9@tcp [ 703.577328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 703.632766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 703.649109] LNet: Added LNI 192.168.202.9@tcp [8/256/0/180] [ 704.180489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.202.[10-15/2]@tcp [ 704.211572] LNet: 33471:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 704.216619] LNet: 33471:0:(router.c:729:lnet_add_route()) Skipped 4 previous similar messages [ 704.223893] LNetError: 33363:0:(lib-lnet.h:1279:lnet_set_route_aliveness()) route to tcp103 through 192.168.202.12@tcp has gone from up to down [ 704.254028] LNetError: 33363:0:(lib-lnet.h:1279:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 704.259312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.202.[10-15/2]@tcp [ 711.917209] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 15:06:09 (1787598369) [ 711.990934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 712.005564] LNet: 34058:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 712.010108] LNetError: 34058:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 712.037875] LNet: Removed LNI 192.168.202.9@tcp [ 712.100867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 712.259237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 712.364650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 712.404668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 712.484430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 712.530172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 712.582395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 712.672611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 712.820528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 712.905448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 713.028437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 713.166769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 713.207966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 713.257431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 713.353141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 713.394328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 713.512491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 713.737651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 713.786255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 713.852667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 713.941736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 713.983573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 714.147745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 714.295048] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 714.355121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 714.410107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 714.497562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 714.533712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 714.622298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 714.706566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 714.746984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 714.814777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 714.885526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 714.953740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 715.096230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 723.115388] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 15:06:20 (1787598380) [ 725.414458] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg209-server [ 727.689300] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 15:06:25 (1787598385) [ 729.818537] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg209-server [ 731.609092] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 15:06:29 (1787598389) [ 731.919847] LNet: 35467:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 731.926351] LNet: 35467:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 11 previous similar messages [ 732.746401] Key type .llcrypt unregistered [ 732.747688] Key type ._llcrypt unregistered [ 738.661451] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 738.855535] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 738.865553] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 738.875207] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 740.418591] Key type ._llcrypt registered [ 740.420022] Key type .llcrypt registered [ 740.603911] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 740.618696] alg: No test for adler32 (adler32-zlib) [ 741.547058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 741.598253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 741.649293] LNet: Added LNI 192.168.202.9@tcp [8/256/0/180] [ 742.525311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 742.570604] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 744.674850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 744.702510] LNet: Removed LNI 10.1.2.3@tcp [ 744.830280] LNet: 36183:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 744.835539] LNetError: 36183:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 744.847361] LNet: Removed LNI 192.168.202.9@tcp [ 745.671784] Key type .llcrypt unregistered [ 745.674106] Key type ._llcrypt unregistered [ 757.070972] Key type ._llcrypt registered [ 757.072904] Key type .llcrypt registered [ 758.873636] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 15:06:56 (1787598416) [ 759.420859] Key type .llcrypt unregistered [ 759.422893] Key type ._llcrypt unregistered [ 765.838178] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 766.036239] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 766.046541] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 766.060242] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 768.692878] Key type ._llcrypt registered [ 768.696655] Key type .llcrypt registered [ 768.886553] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 768.919479] alg: No test for adler32 (adler32-zlib) [ 769.949941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 770.085945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 770.160735] LNet: Added LNI 192.168.202.9@tcp [8/256/0/180] [ 771.111272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 771.140522] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 774.473536] LNet: 37684:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 774.486697] LNetError: 37684:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 774.492401] LNet: Removed LNI 10.1.2.3@tcp [ 774.499789] LNet: Removed LNI 192.168.202.9@tcp [ 775.359776] Key type .llcrypt unregistered [ 775.362460] Key type ._llcrypt unregistered [ 788.002911] Key type ._llcrypt registered [ 788.004393] Key type .llcrypt registered [ 788.850902] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 15:07:27 (1787598447) [ 789.525520] Key type .llcrypt unregistered [ 789.528967] Key type ._llcrypt unregistered [ 796.921462] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 797.061398] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 797.069883] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 797.079868] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 799.776734] Key type ._llcrypt registered [ 799.783110] Key type .llcrypt registered [ 800.029491] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 800.049966] alg: No test for adler32 (adler32-zlib) [ 801.084687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 801.180216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 801.210369] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 802.849899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 802.867824] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 805.507658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 805.540656] LNet: Removed LNI 10.1.2.3@tcp [ 805.563280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 805.604473] LNet: Removed LNI 10.1.2.31@tcp [ 805.655631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 805.710942] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 806.560775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 806.574706] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 807.570350] LNet: 39614:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 807.584843] LNetError: 39614:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg:0 not found [ 807.588184] LNet: Removed LNI 10.1.2.31@tcp [ 807.601017] LNet: Removed LNI 10.1.2.3@tcp [ 808.436782] Key type .llcrypt unregistered [ 808.439766] Key type ._llcrypt unregistered [ 822.119625] Key type ._llcrypt registered [ 822.121782] Key type .llcrypt registered [ 823.760545] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 15:08:01 (1787598481) [ 824.128944] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 824.144441] alg: No test for adler32 (adler32-zlib) [ 825.266765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 825.408574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 825.455648] LNet: Added LNI 192.168.202.9@tcp [8/256/0/180] [ 826.698882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 826.749284] LNet: 40572:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 826.764855] LNetError: 40572:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 826.800525] LNet: Removed LNI 192.168.202.9@tcp [ 826.833661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 826.958058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.202.9@tcp [ 827.019931] LNet: Added LNI 192.168.202.9@tcp [8/256/0/180] [ 828.001365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.202.9@tcp [ 828.043276] LNet: Removed LNI 192.168.202.9@tcp [ 828.142652] LNet: 40707:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 828.895537] Key type .llcrypt unregistered [ 828.898668] Key type ._llcrypt unregistered [ 842.667992] Key type ._llcrypt registered [ 842.670901] Key type .llcrypt registered [ 844.197817] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 15:08:21 (1787598501) [ 846.302527] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg209-server [ 848.202379] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 15:08:26 (1787598506) [ 849.274450] Key type .llcrypt unregistered [ 849.279273] Key type ._llcrypt unregistered [ 856.399949] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 856.591119] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 856.605974] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 856.616695] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 858.204661] Key type ._llcrypt registered [ 858.206854] Key type .llcrypt registered [ 858.435573] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 858.461955] alg: No test for adler32 (adler32-zlib) [ 859.403393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 859.470715] LNet: Added LNI 192.168.202.9@tcp1 [8/256/0/180] [ 860.145894] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 861.947986] LNet: 42539:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 861.957471] LNetError: 42539:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 861.965180] LNet: Removed LNI 192.168.202.9@tcp1 [ 861.975272] LNet: Removed LNI 10.1.2.3@tcp2 [ 862.779638] Key type .llcrypt unregistered [ 862.781850] Key type ._llcrypt unregistered [ 873.832322] Key type ._llcrypt registered [ 873.834447] Key type .llcrypt registered [ 875.281380] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 15:08:53 (1787598533) [ 875.579101] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 875.609811] alg: No test for adler32 (adler32-zlib) [ 876.570071] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 876.676824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 876.732419] LNet: Added LNI 192.168.202.9@tcp [8/256/0/180] [ 877.675606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 877.697205] LNet: 43513:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 877.707639] LNetError: 43513:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 877.723494] LNet: Removed LNI 192.168.202.9@tcp [ 877.736765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 877.799904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 877.826351] LNet: Added LNI 192.168.202.9@tcp [8/256/0/180] [ 884.378963] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 15:09:02 (1787598542) [ 884.717914] LNet: 44104:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 884.725400] LNetError: 44104:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 884.742046] LNet: Removed LNI 192.168.202.9@tcp [ 885.436577] Key type .llcrypt unregistered [ 885.439561] Key type ._llcrypt unregistered [ 891.644234] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 891.856984] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 891.860382] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 891.865309] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 894.582755] Key type ._llcrypt registered [ 894.585266] Key type .llcrypt registered [ 894.776203] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 894.789941] alg: No test for adler32 (adler32-zlib) [ 895.705439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 895.780105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 895.820084] LNet: Added LNI 192.168.202.9@tcp [8/256/0/180] [ 896.533621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 896.564486] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 898.668168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 898.686889] LNet: 44853:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 898.698457] LNetError: 44853:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 898.705427] LNet: Removed LNI 10.1.2.3@tcp [ 898.720113] LNet: Removed LNI 192.168.202.9@tcp [ 898.744543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 898.837805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 898.879238] LNet: Added LNI 192.168.202.9@tcp [8/256/0/180] [ 898.887949] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 899.401965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 899.442274] LNet: 44880:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 899.450853] LNetError: 44880:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 899.462407] LNetError: 44880:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 899.469507] LNet: Removed LNI 10.1.2.3@tcp [ 899.487684] LNet: Removed LNI 192.168.202.9@tcp [ 899.499862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 899.554946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 899.575301] LNet: Added LNI 192.168.202.9@tcp [8/256/0/180] [ 899.580738] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 900.080818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 900.114127] LNet: Added LNI 192.168.202.9@tcp2 [8/256/0/180] [ 900.905428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 900.930801] LNet: 45042:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 900.941577] LNetError: 45042:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 900.950788] LNetError: 45042:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 900.957808] LNet: Removed LNI 10.1.2.3@tcp [ 900.960552] LNet: Removed LNI 192.168.202.9@tcp [ 900.975400] LNet: Removed LNI 192.168.202.9@tcp2 [ 900.993059] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 901.086298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 901.157980] LNet: Added LNI 192.168.202.9@tcp [8/256/0/180] [ 901.168348] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 901.175247] LNet: Added LNI 192.168.202.9@tcp2 [8/256/0/180] [ 908.917379] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 15:09:27 (1787598567) [ 909.312849] LNet: 45642:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 909.325338] LNetError: 45642:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 909.335038] LNetError: 45642:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 909.340829] LNet: Removed LNI 10.1.2.3@tcp [ 909.342309] LNet: Removed LNI 192.168.202.9@tcp [ 909.357494] LNet: Removed LNI 192.168.202.9@tcp2 [ 909.784513] Key type .llcrypt unregistered [ 909.785697] Key type ._llcrypt unregistered [ 915.885884] Key type ._llcrypt registered [ 915.887712] Key type .llcrypt registered [ 916.045202] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 916.085610] alg: No test for adler32 (adler32-zlib) [ 917.100532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 917.167499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 917.205851] LNet: Added LNI 192.168.202.9@tcp [8/256/0/180] [ 918.365733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 918.382950] LNet: 46040:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 918.388758] LNetError: 46040:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 918.405314] LNet: Removed LNI 192.168.202.9@tcp [ 918.421808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 918.468888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 918.488965] LNet: Added LNI 192.168.202.9@tcp [8/256/0/180] [ 919.174591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 919.207619] LNet: 46070:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 919.223444] LNetError: 46070:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 919.237762] LNet: Removed LNI 192.168.202.9@tcp [ 919.261952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 919.319124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 919.349151] LNet: Added LNI 192.168.202.9@tcp [8/256/0/180] [ 919.734395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 919.760646] LNet: Removed LNI 192.168.202.9@tcp [ 919.776483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 919.828993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 919.850660] LNet: Added LNI 192.168.202.9@tcp [8/256/0/180] [ 920.318255] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 920.527352] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 920.535869] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 920.539843] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 922.875819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 922.899603] LNet: 46179:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 922.904822] LNet: 46179:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 922.910524] LNetError: 46179:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 922.916570] LNetError: 46179:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 922.931296] LNet: Removed LNI 192.168.202.9@tcp [ 922.966670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 923.026364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 923.057711] LNet: Added LNI 192.168.202.9@tcp [8/256/0/180] [ 931.098768] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 15:09:48 (1787598588) [ 931.211159] LNet: 46767:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 931.220446] LNetError: 46767:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 931.234879] LNet: Removed LNI 192.168.202.9@tcp [ 932.200940] Key type .llcrypt unregistered [ 932.206141] Key type ._llcrypt unregistered [ 939.712342] Key type ._llcrypt registered [ 939.717556] Key type .llcrypt registered [ 939.928442] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 939.946486] alg: No test for adler32 (adler32-zlib) [ 940.977430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 941.122671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 941.176854] LNet: Added LNI 192.168.202.9@tcp [8/256/0/180] [ 942.348440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 942.370269] LNet: 47179:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 942.376942] LNetError: 47179:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 942.392893] LNet: Removed LNI 192.168.202.9@tcp [ 942.430447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 942.527281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 942.570955] LNet: Added LNI 192.168.202.9@tcp [8/256/0/180] [ 950.450361] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 15:10:08 (1787598608) [ 950.562823] LNet: 47763:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 950.581096] LNetError: 47763:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 950.616339] LNet: Removed LNI 192.168.202.9@tcp [ 951.661492] Key type .llcrypt unregistered [ 951.665555] Key type ._llcrypt unregistered [ 958.341343] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 958.615322] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 958.629057] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 958.643023] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 961.516351] Key type ._llcrypt registered [ 961.518588] Key type .llcrypt registered [ 961.786988] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 961.856986] alg: No test for adler32 (adler32-zlib) [ 962.922263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 963.055186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 963.127097] LNet: Added LNI 192.168.202.9@tcp [8/256/0/180] [ 964.311227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 964.349983] LNet: 48223:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 964.355980] LNetError: 48223:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 964.371283] LNet: Removed LNI 192.168.202.9@tcp [ 964.409960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 964.478469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 964.508248] LNet: Added LNI 192.168.202.9@tcp [8/256/0/180] [ 972.641262] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 15:10:30 (1787598630) [ 972.803230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 972.861501] LNet: 48813:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 972.873950] LNetError: 48813:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 972.911549] LNet: Removed LNI 192.168.202.9@tcp [ 972.952299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 973.139873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 981.165805] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 15:10:38 (1787598638) [ 981.263559] LNet: 49395:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 982.199672] Key type .llcrypt unregistered [ 982.204082] Key type ._llcrypt unregistered [ 988.960861] Key type ._llcrypt registered [ 988.966648] Key type .llcrypt registered [ 989.369813] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 989.407289] alg: No test for adler32 (adler32-zlib) [ 990.511938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 990.606757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 990.646338] LNet: Added LNI 192.168.202.9@tcp [8/256/0/180] [ 991.848146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 991.877812] LNet: 49805:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 991.887881] LNetError: 49805:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 991.905844] LNet: Removed LNI 192.168.202.9@tcp [ 991.938685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 992.015650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 992.049383] LNet: Added LNI 192.168.202.9@tcp [8/256/0/180] [ 1000.254262] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 15:10:58 (1787598658) [ 1000.891857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1000.982147] LNet: 50400:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1001.015459] LNetError: 50400:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1001.040639] LNet: Removed LNI 192.168.202.9@tcp [ 1001.077975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1001.159518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 1009.044564] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 15:11:06 (1787598666) [ 1010.844517] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 1012.990390] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 15:11:10 (1787598670) [ 1014.716659] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 1016.840853] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 15:11:14 (1787598674) [ 1018.758241] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 1020.985661] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 15:11:18 (1787598678) [ 1022.842455] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 1024.793280] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 15:11:22 (1787598682) [ 1026.299856] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 1028.211494] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing ========================================================== 15:11:26 (1787598686) [ 1028.637947] LNet: 52455:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1029.462278] Key type .llcrypt unregistered [ 1029.463538] Key type ._llcrypt unregistered [ 1037.083094] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1037.229577] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1037.241862] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1037.256622] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1040.033016] Key type ._llcrypt registered [ 1040.035911] Key type .llcrypt registered [ 1040.276415] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1040.294590] alg: No test for adler32 (adler32-zlib) [ 1041.292252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1041.391850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1041.411202] LNet: Added LNI 192.168.202.9@tcp [8/512/0/180] [ 1041.419887] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 1041.424377] LNet: 52781:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1041.529608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1041.549346] LNet: 52800:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1041.554788] LNetError: 52800:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1041.569000] LNet: Removed LNI 10.1.2.3@tcp [ 1041.570496] LNetError: 52800:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1041.589950] LNet: Removed LNI 192.168.202.9@tcp [ 1041.608720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1041.694951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1041.717401] LNet: Added LNI 192.168.202.9@tcp [8/256/0/180] [ 1041.795662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1041.816710] LNet: 52824:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1041.823741] LNetError: 52824:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1041.845906] LNet: Removed LNI 192.168.202.9@tcp [ 1041.864637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1041.925870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1041.955230] LNet: Added LNI 192.168.202.9@tcp [8/256/0/180] [ 1041.960162] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1041.962090] LNet: 52832:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1042.133612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1042.181827] LNet: 52850:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1042.194970] LNetError: 52850:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1042.201210] LNet: Removed LNI 10.1.2.3@tcp [ 1042.203764] LNetError: 52850:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1042.222709] LNet: Removed LNI 192.168.202.9@tcp [ 1042.258592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1042.376410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1042.406924] LNet: Added LNI 192.168.202.9@tcp [8/256/0/180] [ 1042.411951] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1042.415428] LNet: 52858:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1042.731691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1042.767424] LNet: 52884:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1042.777514] LNetError: 52884:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1042.788360] LNet: Removed LNI 10.1.2.3@tcp [ 1042.794421] LNetError: 52884:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1042.816807] LNet: Removed LNI 192.168.202.9@tcp [ 1042.856220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1042.987493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1043.036750] LNet: Added LNI 192.168.202.9@tcp [8/256/0/182] [ 1043.042444] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 1043.202915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1043.240888] LNet: 52912:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1043.251883] LNetError: 52912:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1043.258644] LNet: Removed LNI 10.1.2.3@tcp [ 1043.260133] LNetError: 52912:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1043.278900] LNet: Removed LNI 192.168.202.9@tcp [ 1043.305904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1043.386606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1043.410487] LNet: Added LNI 192.168.202.9@tcp [8/256/0/182] [ 1043.414421] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 1043.417526] LNet: 52920:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1043.489369] LNet: Removed LNI 10.1.2.3@tcp [ 1043.497042] LNet: Removed LNI 192.168.202.9@tcp [ 1044.305793] Key type .llcrypt unregistered [ 1044.312057] Key type ._llcrypt unregistered [ 1056.494454] Key type ._llcrypt registered [ 1056.498875] Key type .llcrypt registered [ 1057.991404] Lustre: DEBUG MARKER: == sanity-lnet test 180: FastReg map error should not leak descriptors ========================================================== 15:11:55 (1787598715) [ 1060.126668] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_180 Need o2ib NETTYPE [ 1062.191775] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 15:11:59 (1787598719) [ 1062.746767] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1062.779231] alg: No test for adler32 (adler32-zlib) [ 1063.717687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1063.806486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1063.827789] LNet: Added LNI 192.168.202.9@tcp [8/256/0/180] [ 1064.648756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.202.9 [ 1064.665298] LNet: 54189:0:(config.c:1738:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1064.673355] LNet: Removed LNI 192.168.202.9@tcp [ 1071.169316] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 15:12:09 (1787598729) [ 1071.599396] LNet: 54759:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1071.604497] LNetError: 54759:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1071.625699] LNet: Removed LNI 192.168.202.9@tcp [ 1072.276500] Key type .llcrypt unregistered [ 1072.279097] Key type ._llcrypt unregistered [ 1080.561529] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1080.815846] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1083.737315] Key type ._llcrypt registered [ 1083.740573] Key type .llcrypt registered [ 1083.955162] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1083.988967] alg: No test for adler32 (adler32-zlib) [ 1085.293275] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1092.122368] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 15:12:29 (1787598749) [ 1092.416966] LNet: 55677:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1092.426802] LNetError: 55677:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1092.448973] LNet: Removed LNI 10.1.2.3@tcp [ 1093.366927] Key type .llcrypt unregistered [ 1093.369429] Key type ._llcrypt unregistered [ 1102.124776] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1102.341553] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1102.346126] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1102.349466] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1105.199660] Key type ._llcrypt registered [ 1105.202131] Key type .llcrypt registered [ 1105.353749] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1105.361613] alg: No test for adler32 (adler32-zlib) [ 1106.295599] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1113.154414] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 15:12:51 (1787598771) [ 1114.523231] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 1116.090942] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 15:12:54 (1787598774) [ 1116.657532] LNet: 56867:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1116.665950] LNetError: 56867:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1116.691694] LNet: Removed LNI 10.1.2.3@tcp [ 1117.377850] Key type .llcrypt unregistered [ 1117.380672] Key type ._llcrypt unregistered [ 1123.545225] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1123.784537] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1123.787587] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1123.796705] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1126.455470] Key type ._llcrypt registered [ 1126.457472] Key type .llcrypt registered [ 1126.617507] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1126.626727] alg: No test for adler32 (adler32-zlib) [ 1127.492106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1127.546261] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1135.627214] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 15:13:13 (1787598793) [ 1136.156448] LNet: 58101:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1136.164117] LNetError: 58101:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1136.177129] LNet: Removed LNI 10.1.2.3@tcp [ 1137.203696] Key type .llcrypt unregistered [ 1137.210414] Key type ._llcrypt unregistered [ 1145.233906] Key type ._llcrypt registered [ 1145.235950] Key type .llcrypt registered [ 1145.612820] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1145.631149] alg: No test for adler32 (adler32-zlib) [ 1146.688739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1146.724524] LNet: Added LNI 192.168.202.9@tcp [8/256/0/180] [ 1156.920389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.109@tcp [ 1157.834395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.9@tcp -d 192.168.202.109@tcp -m GET -e local_interrupt -r 1 [ 1157.919424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.9@tcp -d 192.168.202.9@tcp -m GET -e local_interrupt -r 1 [ 1157.992183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.109@tcp [ 1158.013638] LNet: There was an unexpected network error while writing to 192.168.202.109: rc = -22 [ 1158.410582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1158.461471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1158.518848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1158.569832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1158.671309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1158.759679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1158.831800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1158.893317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1158.925515] LNet: 58544:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1158.948110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1159.012261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1159.020111] LNet: 1 local NIs in recovery (showing 1): 192.168.202.9@tcp [ 1159.023661] LNet: There was an unexpected network error while writing to 192.168.202.9: rc = -22 [ 1159.069297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1159.101648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1159.143212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1159.179721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1159.225655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1160.063841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.9@tcp -d 192.168.202.109@tcp -m GET -e local_dropped -r 1 [ 1160.109624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.9@tcp -d 192.168.202.9@tcp -m GET -e local_dropped -r 1 [ 1160.156892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.109@tcp [ 1160.179538] LNet: There was an unexpected network error while writing to 192.168.202.109: rc = -22 [ 1160.453866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1160.501314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1160.554182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1160.601990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1160.657266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1160.728480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1160.779872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1160.823256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1160.855164] LNet: 58654:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1160.870217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1160.915974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1160.969982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1161.011842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1161.049245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1161.071884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1161.111180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1161.608395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.9@tcp -d 192.168.202.109@tcp -m GET -e local_aborted -r 1 [ 1161.636741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.9@tcp -d 192.168.202.9@tcp -m GET -e local_aborted -r 1 [ 1161.668433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.109@tcp [ 1161.942346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1161.977839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1162.015655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1162.046666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1162.083923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1162.106940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1162.127024] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1162.153322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1162.171644] LNet: 58763:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1162.189524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1162.217907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1162.247794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1162.282455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1162.323288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1162.378972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1162.434478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1163.122300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.9@tcp -d 192.168.202.109@tcp -m GET -e local_no_route -r 1 [ 1163.170810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.9@tcp -d 192.168.202.9@tcp -m GET -e local_no_route -r 1 [ 1163.210833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.109@tcp [ 1163.239567] LNet: There was an unexpected network error while writing to 192.168.202.109: rc = -22 [ 1163.248086] LNet: Skipped 1 previous similar message [ 1163.556700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1163.578592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1163.607471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1163.633674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1163.672872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1163.718680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1163.764508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1163.807841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1163.869453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1163.968353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1164.075569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1164.147731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1164.181745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1164.215754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1164.290976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1165.028734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.9@tcp -d 192.168.202.109@tcp -m GET -e local_timeout -r 1 [ 1165.083262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.9@tcp -d 192.168.202.9@tcp -m GET -e local_timeout -r 1 [ 1165.125639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.109@tcp [ 1165.425299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1165.468633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1165.513393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1165.545377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1165.579787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1165.614918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1165.651807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1165.686418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1165.698332] LNet: 58982:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1165.701467] LNet: 58982:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1165.714591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1165.744505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1165.791638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1165.835373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1165.869495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1165.903962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1165.972734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1166.584726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.9@tcp -d 192.168.202.109@tcp -m GET -e local_error -r 1 [ 1166.626375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.9@tcp -d 192.168.202.9@tcp -m GET -e local_error -r 1 [ 1166.673814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.109@tcp [ 1166.960320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1167.005767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1167.069421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1167.115691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1167.141867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1167.181669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1167.236036] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1167.291442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1167.336683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1167.388039] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1167.446892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1167.495687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1167.593988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1167.664974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1167.721744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1168.049480] LNet: 59136:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1168.056830] LNetError: 59136:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1168.074717] LNet: Removed LNI 192.168.202.9@tcp [ 1168.500504] Key type .llcrypt unregistered [ 1168.502580] Key type ._llcrypt unregistered [ 1188.105659] Key type ._llcrypt registered [ 1188.108262] Key type .llcrypt registered [ 1189.435941] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 15:14:07 (1787598847) [ 1190.371252] Key type .llcrypt unregistered [ 1190.373638] Key type ._llcrypt unregistered [ 1198.069205] Key type ._llcrypt registered [ 1198.075792] Key type .llcrypt registered [ 1198.450630] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1198.492586] alg: No test for adler32 (adler32-zlib) [ 1199.581988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1199.667170] LNet: Added LNI 192.168.202.9@tcp [8/256/0/180] [ 1208.656410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.109@tcp [ 1221.092760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1221.156606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1221.182610] LNet: Added LNI 192.168.202.9@tcp1 [8/256/0/180] [ 1222.376711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.9@tcp -d 192.168.202.109@tcp -m GET -e local_interrupt -r 1 [ 1222.431208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.9@tcp -d 192.168.202.109@tcp1 -m GET -e local_interrupt -r 1 [ 1222.473152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.9@tcp -d 192.168.202.9@tcp -m GET -e local_interrupt -r 1 [ 1222.545300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.9@tcp -d 192.168.202.9@tcp1 -m GET -e local_interrupt -r 1 [ 1222.596952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.9@tcp1 -d 192.168.202.109@tcp -m GET -e local_interrupt -r 1 [ 1222.640675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.9@tcp1 -d 192.168.202.109@tcp1 -m GET -e local_interrupt -r 1 [ 1222.680273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.9@tcp1 -d 192.168.202.9@tcp -m GET -e local_interrupt -r 1 [ 1222.736988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.9@tcp1 -d 192.168.202.9@tcp1 -m GET -e local_interrupt -r 1 [ 1222.805893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.109@tcp [ 1222.827529] LNet: There was an unexpected network error while writing to 192.168.202.109: rc = -22 [ 1222.833690] LNet: 1 local NIs in recovery (showing 1): 192.168.202.9@tcp [ 1223.083574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1223.140166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1223.178909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1223.210889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1223.237197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1223.262387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1223.290744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1223.331293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1223.353387] LNet: 60611:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1223.385372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1223.443073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1223.492827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1223.521554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1223.558370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1223.588696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1223.633233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1224.245133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.9@tcp -d 192.168.202.109@tcp -m GET -e local_dropped -r 1 [ 1224.285383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.9@tcp -d 192.168.202.109@tcp1 -m GET -e local_dropped -r 1 [ 1224.345386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.9@tcp -d 192.168.202.9@tcp -m GET -e local_dropped -r 1 [ 1224.397035] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.9@tcp -d 192.168.202.9@tcp1 -m GET -e local_dropped -r 1 [ 1224.439938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.9@tcp1 -d 192.168.202.109@tcp -m GET -e local_dropped -r 1 [ 1224.501919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.9@tcp1 -d 192.168.202.109@tcp1 -m GET -e local_dropped -r 1 [ 1224.578125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.9@tcp1 -d 192.168.202.9@tcp -m GET -e local_dropped -r 1 [ 1224.618337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.9@tcp1 -d 192.168.202.9@tcp1 -m GET -e local_dropped -r 1 [ 1224.650320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.109@tcp [ 1224.666228] LNet: There was an unexpected network error while writing to 192.168.202.109: rc = -22 [ 1224.670219] LNet: Skipped 2 previous similar messages [ 1224.896969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1224.963687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1225.004992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1225.028846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1225.053721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1225.097900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1225.169449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1225.220026] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1225.240076] LNet: 60732:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1225.262475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1225.297950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1225.333202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1225.365728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1225.397614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1225.431956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1225.464934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1226.103393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.9@tcp -d 192.168.202.109@tcp -m GET -e local_aborted -r 1 [ 1226.167332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.9@tcp -d 192.168.202.109@tcp1 -m GET -e local_aborted -r 1 [ 1226.210542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.9@tcp -d 192.168.202.9@tcp -m GET -e local_aborted -r 1 [ 1226.248158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.9@tcp -d 192.168.202.9@tcp1 -m GET -e local_aborted -r 1 [ 1226.292363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.9@tcp1 -d 192.168.202.109@tcp -m GET -e local_aborted -r 1 [ 1226.331613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.9@tcp1 -d 192.168.202.109@tcp1 -m GET -e local_aborted -r 1 [ 1226.355725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.9@tcp1 -d 192.168.202.9@tcp -m GET -e local_aborted -r 1 [ 1226.393591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.9@tcp1 -d 192.168.202.9@tcp1 -m GET -e local_aborted -r 1 [ 1226.443365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.109@tcp [ 1226.474734] LNet: There was an unexpected network error while writing to 192.168.202.109: rc = -22 [ 1226.485529] LNet: Skipped 2 previous similar messages [ 1226.802817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1226.856931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1226.877293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1226.905115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1226.956056] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1227.004151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1227.048194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1227.098945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1227.135922] LNet: 60854:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1227.165795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1227.195918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1227.230389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1227.254980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1227.302951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1227.343766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1227.379817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1228.051631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.9@tcp -d 192.168.202.109@tcp -m GET -e local_no_route -r 1 [ 1228.109110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.9@tcp -d 192.168.202.109@tcp1 -m GET -e local_no_route -r 1 [ 1228.161668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.9@tcp -d 192.168.202.9@tcp -m GET -e local_no_route -r 1 [ 1228.219223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.9@tcp -d 192.168.202.9@tcp1 -m GET -e local_no_route -r 1 [ 1228.278566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.9@tcp1 -d 192.168.202.109@tcp -m GET -e local_no_route -r 1 [ 1228.356423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.9@tcp1 -d 192.168.202.109@tcp1 -m GET -e local_no_route -r 1 [ 1228.402855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.9@tcp1 -d 192.168.202.9@tcp -m GET -e local_no_route -r 1 [ 1228.477468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.9@tcp1 -d 192.168.202.9@tcp1 -m GET -e local_no_route -r 1 [ 1228.516513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.109@tcp [ 1228.543849] LNet: There was an unexpected network error while writing to 192.168.202.109: rc = -22 [ 1228.547074] LNet: Skipped 2 previous similar messages [ 1228.809278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1228.840212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1228.899640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1228.944916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1228.995507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1229.032930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1229.060922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1229.093712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1229.130640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1229.181973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1229.234591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1229.305974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1229.346174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1229.379433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1229.409832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1229.846657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.9@tcp -d 192.168.202.109@tcp -m GET -e local_timeout -r 1 [ 1229.876401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.9@tcp -d 192.168.202.109@tcp1 -m GET -e local_timeout -r 1 [ 1229.918648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.9@tcp -d 192.168.202.9@tcp -m GET -e local_timeout -r 1 [ 1229.942529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.9@tcp -d 192.168.202.9@tcp1 -m GET -e local_timeout -r 1 [ 1229.984910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.9@tcp1 -d 192.168.202.109@tcp -m GET -e local_timeout -r 1 [ 1230.027798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.9@tcp1 -d 192.168.202.109@tcp1 -m GET -e local_timeout -r 1 [ 1230.061970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.9@tcp1 -d 192.168.202.9@tcp -m GET -e local_timeout -r 1 [ 1230.100763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.9@tcp1 -d 192.168.202.9@tcp1 -m GET -e local_timeout -r 1 [ 1230.138293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.109@tcp [ 1230.371174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1230.396595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1230.438337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1230.475129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1230.532701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1230.596890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1230.632795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1230.719321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1230.759782] LNet: 61096:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1230.768049] LNet: 61096:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1230.794847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1230.832488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1230.856487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1230.891618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1230.930804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1231.001982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1231.055582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1231.858827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.9@tcp -d 192.168.202.109@tcp -m GET -e local_error -r 1 [ 1231.944897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.9@tcp -d 192.168.202.109@tcp1 -m GET -e local_error -r 1 [ 1232.003741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.9@tcp -d 192.168.202.9@tcp -m GET -e local_error -r 1 [ 1232.103614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.9@tcp -d 192.168.202.9@tcp1 -m GET -e local_error -r 1 [ 1232.169405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.9@tcp1 -d 192.168.202.109@tcp -m GET -e local_error -r 1 [ 1232.220972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.9@tcp1 -d 192.168.202.109@tcp1 -m GET -e local_error -r 1 [ 1232.286988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.9@tcp1 -d 192.168.202.9@tcp -m GET -e local_error -r 1 [ 1232.309768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.9@tcp1 -d 192.168.202.9@tcp1 -m GET -e local_error -r 1 [ 1232.338917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.109@tcp [ 1232.531876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1232.561596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1232.586511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1232.611870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1232.643541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1232.672478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1232.703489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1232.734309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1232.757923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1232.783805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1232.851912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1232.923834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1232.989894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1233.038899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1233.097397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1239.427881] LNet: 61298:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1239.444591] LNetError: 61298:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1239.450757] LNet: Removed LNI 192.168.202.9@tcp [ 1239.461889] LNet: Removed LNI 192.168.202.9@tcp1 [ 1239.837735] Key type .llcrypt unregistered [ 1239.839379] Key type ._llcrypt unregistered [ 1255.241416] Key type ._llcrypt registered [ 1255.243514] Key type .llcrypt registered [ 1256.552969] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 15:15:14 (1787598914) [ 1257.740709] Key type .llcrypt unregistered [ 1257.744669] Key type ._llcrypt unregistered [ 1265.087850] Key type ._llcrypt registered [ 1265.090315] Key type .llcrypt registered [ 1265.299093] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1265.319422] alg: No test for adler32 (adler32-zlib) [ 1266.278095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1266.300397] LNet: Added LNI 192.168.202.9@tcp [8/256/0/180] [ 1275.290476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.109@tcp [ 1276.243098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.9@tcp -d 192.168.202.109@tcp -m GET -e remote_dropped -r 1 [ 1276.281545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.9@tcp -d 192.168.202.9@tcp -m GET -e remote_dropped -r 1 [ 1276.321513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.109@tcp [ 1276.334105] LNet: There was an unexpected network error while writing to 192.168.202.109: rc = -22 [ 1276.599483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1276.640512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1276.682380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1276.708636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1276.733217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1276.761518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1276.790340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1276.829524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1276.849458] LNet: 62551:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1276.866494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1276.900118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1276.951367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1277.004357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1277.066812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1277.144638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1277.191749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1277.963791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.9@tcp -d 192.168.202.109@tcp -m GET -e remote_error -r 1 [ 1277.988661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.9@tcp -d 192.168.202.9@tcp -m GET -e remote_error -r 1 [ 1278.023374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.109@tcp [ 1278.050023] LNet: There was an unexpected network error while writing to 192.168.202.109: rc = -22 [ 1278.350744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1278.385582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1278.415453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1278.444481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1278.479774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1278.513561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1278.546191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1278.590478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1278.611508] LNet: 62660:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1278.635416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1278.677863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1278.720628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1278.767486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1278.821254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1278.879189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1278.941309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1279.597486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.9@tcp -d 192.168.202.109@tcp -m GET -e remote_timeout -r 1 [ 1279.643289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.9@tcp -d 192.168.202.9@tcp -m GET -e remote_timeout -r 1 [ 1279.673989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.109@tcp [ 1279.688889] LNet: There was an unexpected network error while writing to 192.168.202.109: rc = -22 [ 1279.879407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1279.911671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1279.941142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1279.970372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1279.995635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1280.020375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1280.042599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1280.072576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1280.089555] LNet: 62769:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1280.105956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1280.140875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1280.171994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1280.205619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1280.233484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1280.263787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1280.289105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1280.542608] LNet: 62814:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1280.551757] LNetError: 62814:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1280.582065] LNet: Removed LNI 192.168.202.9@tcp [ 1280.971582] Key type .llcrypt unregistered [ 1280.975575] Key type ._llcrypt unregistered [ 1300.132092] Key type ._llcrypt registered [ 1300.134630] Key type .llcrypt registered [ 1301.887934] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 15:15:59 (1787598959) [ 1303.328226] Key type .llcrypt unregistered [ 1303.330644] Key type ._llcrypt unregistered [ 1310.274441] Key type ._llcrypt registered [ 1310.279482] Key type .llcrypt registered [ 1310.536181] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1310.562357] alg: No test for adler32 (adler32-zlib) [ 1311.547966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1311.612490] LNet: Added LNI 192.168.202.9@tcp [8/256/0/180] [ 1319.593105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.109@tcp [ 1327.290526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1327.346632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1327.382584] LNet: Added LNI 192.168.202.9@tcp1 [8/256/0/180] [ 1328.818796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.9@tcp -d 192.168.202.109@tcp -m GET -e remote_dropped -r 1 [ 1328.888719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.9@tcp -d 192.168.202.109@tcp1 -m GET -e remote_dropped -r 1 [ 1328.949709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.9@tcp -d 192.168.202.9@tcp -m GET -e remote_dropped -r 1 [ 1329.001215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.9@tcp -d 192.168.202.9@tcp1 -m GET -e remote_dropped -r 1 [ 1329.037845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.9@tcp1 -d 192.168.202.109@tcp -m GET -e remote_dropped -r 1 [ 1329.121620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.9@tcp1 -d 192.168.202.109@tcp1 -m GET -e remote_dropped -r 1 [ 1329.162385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.9@tcp1 -d 192.168.202.9@tcp -m GET -e remote_dropped -r 1 [ 1329.210885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.9@tcp1 -d 192.168.202.9@tcp1 -m GET -e remote_dropped -r 1 [ 1329.247513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.109@tcp [ 1329.260809] LNet: There was an unexpected network error while writing to 192.168.202.109: rc = -22 [ 1329.264644] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.109@tcp [ 1329.445857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1329.476908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1329.511443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1329.554364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1329.600902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1329.666691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1329.731818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1329.758966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1329.785897] LNet: 64284:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1329.813847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1329.854684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1329.886606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1329.921185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1329.962567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1329.998696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1330.035814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1330.090078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1330.618496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.9@tcp -d 192.168.202.109@tcp -m GET -e remote_error -r 1 [ 1330.659604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.9@tcp -d 192.168.202.109@tcp1 -m GET -e remote_error -r 1 [ 1330.697354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.9@tcp -d 192.168.202.9@tcp -m GET -e remote_error -r 1 [ 1330.740758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.9@tcp -d 192.168.202.9@tcp1 -m GET -e remote_error -r 1 [ 1330.777321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.9@tcp1 -d 192.168.202.109@tcp -m GET -e remote_error -r 1 [ 1330.825345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.9@tcp1 -d 192.168.202.109@tcp1 -m GET -e remote_error -r 1 [ 1330.862245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.9@tcp1 -d 192.168.202.9@tcp -m GET -e remote_error -r 1 [ 1330.902576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.9@tcp1 -d 192.168.202.9@tcp1 -m GET -e remote_error -r 1 [ 1330.940825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.109@tcp [ 1330.954810] LNet: There was an unexpected network error while writing to 192.168.202.109: rc = -22 [ 1330.968545] LNet: Skipped 2 previous similar messages [ 1331.332904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1331.373710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1331.459173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1331.524828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1331.583079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1331.654093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1331.692425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1331.741863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1331.757810] LNet: 64407:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1331.787170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1331.855369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1331.914226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1331.974885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1332.020871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1332.060604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1332.099557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1332.150361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1332.740641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.9@tcp -d 192.168.202.109@tcp -m GET -e remote_timeout -r 1 [ 1332.780557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.9@tcp -d 192.168.202.109@tcp1 -m GET -e remote_timeout -r 1 [ 1332.822317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.9@tcp -d 192.168.202.9@tcp -m GET -e remote_timeout -r 1 [ 1332.855391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.9@tcp -d 192.168.202.9@tcp1 -m GET -e remote_timeout -r 1 [ 1332.883756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.9@tcp1 -d 192.168.202.109@tcp -m GET -e remote_timeout -r 1 [ 1332.923254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.9@tcp1 -d 192.168.202.109@tcp1 -m GET -e remote_timeout -r 1 [ 1332.956142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.9@tcp1 -d 192.168.202.9@tcp -m GET -e remote_timeout -r 1 [ 1332.995367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.9@tcp1 -d 192.168.202.9@tcp1 -m GET -e remote_timeout -r 1 [ 1333.050899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.109@tcp [ 1333.067929] LNet: There was an unexpected network error while writing to 192.168.202.109: rc = -22 [ 1333.071318] LNet: Skipped 1 previous similar message [ 1333.283568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1333.307712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1333.337777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1333.394252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1333.423697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1333.459272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1333.509639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1333.568949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1333.588651] LNet: 64531:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1333.605341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1333.639989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1333.691399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1333.756893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1333.795933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1333.862114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1333.905948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1333.958362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1338.487975] LNet: 64613:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1338.495263] LNetError: 64613:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1338.505830] LNet: Removed LNI 192.168.202.9@tcp [ 1338.518084] LNet: Removed LNI 192.168.202.9@tcp1 [ 1338.912557] Key type .llcrypt unregistered [ 1338.915368] Key type ._llcrypt unregistered [ 1356.166595] Key type ._llcrypt registered [ 1356.169555] Key type .llcrypt registered [ 1357.724586] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 15:16:55 (1787599015) [ 1358.799840] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 1360.370761] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 15:16:58 (1787599018) [ 1361.446291] Key type .llcrypt unregistered [ 1361.448782] Key type ._llcrypt unregistered [ 1368.463368] Key type ._llcrypt registered [ 1368.465666] Key type .llcrypt registered [ 1368.766556] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1368.786744] alg: No test for adler32 (adler32-zlib) [ 1369.721635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1369.746428] LNet: Added LNI 192.168.202.9@tcp [8/256/0/180] [ 1376.941726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.109@tcp [ 1377.706961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.9@tcp -d 192.168.202.109@tcp -m GET -e network_timeout -r 1 [ 1377.767076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.9@tcp -d 192.168.202.9@tcp -m GET -e network_timeout -r 1 [ 1377.840193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.109@tcp [ 1377.862362] LNet: There was an unexpected network error while writing to 192.168.202.109: rc = -22 [ 1377.955801] LNet: 1 local NIs in recovery (showing 1): 192.168.202.9@tcp [ 1378.236516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1378.285413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1378.331246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1378.374954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1378.409596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1378.451456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1378.488225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1378.531884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1378.562414] LNet: 66161:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1378.587671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1378.641352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1378.687380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1378.735407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1378.779795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1378.831358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1379.251596] LNet: 66204:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1379.258532] LNetError: 66204:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1380.335475] LNet: Removed LNI 192.168.202.9@tcp [ 1380.884283] Key type .llcrypt unregistered [ 1380.886994] Key type ._llcrypt unregistered [ 1404.910550] Key type ._llcrypt registered [ 1404.913441] Key type .llcrypt registered [ 1405.318483] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1405.349837] alg: No test for adler32 (adler32-zlib) [ 1406.338566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1406.366266] LNet: Added LNI 192.168.202.9@tcp [8/256/0/180] [ 1414.274940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.109@tcp [ 1425.138857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1425.222660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1425.247702] LNet: Added LNI 192.168.202.9@tcp1 [8/256/0/180] [ 1427.159381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.9@tcp -d 192.168.202.109@tcp -m GET -e network_timeout -r 1 [ 1427.223454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.9@tcp -d 192.168.202.109@tcp1 -m GET -e network_timeout -r 1 [ 1427.318246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.9@tcp -d 192.168.202.9@tcp -m GET -e network_timeout -r 1 [ 1427.408429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.9@tcp -d 192.168.202.9@tcp1 -m GET -e network_timeout -r 1 [ 1427.489524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.9@tcp1 -d 192.168.202.109@tcp -m GET -e network_timeout -r 1 [ 1427.598060] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.9@tcp1 -d 192.168.202.109@tcp1 -m GET -e network_timeout -r 1 [ 1427.683403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.9@tcp1 -d 192.168.202.9@tcp -m GET -e network_timeout -r 1 [ 1427.763658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.9@tcp1 -d 192.168.202.9@tcp1 -m GET -e network_timeout -r 1 [ 1427.862149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.109@tcp [ 1427.916831] LNet: There was an unexpected network error while writing to 192.168.202.109: rc = -22 [ 1428.322085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1428.388149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1428.438354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1428.496498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1428.554316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1428.592040] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1428.665834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1428.739408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1428.776594] LNet: 67089:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1428.815968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1428.896095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1428.898072] LNet: 1 local NIs in recovery (showing 1): 192.168.202.9@tcp [ 1428.898583] LNet: There was an unexpected network error while writing to 192.168.202.109: rc = -22 [ 1428.945334] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.109@tcp [ 1428.955751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1429.004624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1429.049507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1429.099376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1434.533124] LNet: There was an unexpected network error while writing to 192.168.202.109: rc = -22 [ 1434.548076] LNet: Skipped 1 previous similar message [ 1435.000785] LNet: 67166:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1435.024565] LNetError: 67166:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1435.038920] LNet: Removed LNI 192.168.202.9@tcp [ 1435.058642] LNet: Removed LNI 192.168.202.9@tcp1 [ 1435.541741] Key type .llcrypt unregistered [ 1435.546193] Key type ._llcrypt unregistered [ 1452.737298] Key type ._llcrypt registered [ 1452.739113] Key type .llcrypt registered [ 1454.210292] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 15:18:31 (1787599111) [ 1454.503512] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1454.524649] alg: No test for adler32 (adler32-zlib) [ 1455.465309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1455.516751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1455.534115] LNet: Added LNI 192.168.202.9@tcp [8/256/0/180] [ 1456.216940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1456.235282] LNet: Added LNI 192.168.202.9@tcp1 [8/256/0/180] [ 1457.015216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.9@tcp [ 1457.093647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1457.177543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1457.222679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1457.272709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1457.306582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1457.340293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.202.9@tcp [ 1457.358210] LNetError: 68268:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.202.9@tcp to recovery [ 1457.567434] LNet: 1 local NIs in recovery (showing 1): 192.168.202.9@tcp [ 1457.568472] LNet: There was an unexpected network error while writing to 192.168.202.9: rc = -22 [ 1459.622637] LNet: There was an unexpected network error while writing to 192.168.202.9: rc = -22 [ 1459.768433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1463.711808] LNet: There was an unexpected network error while writing to 192.168.202.9: rc = -22 [ 1464.499219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1464.533904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1464.676114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1464.730397] LNet: 68375:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1464.734765] LNetError: 68375:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1464.739643] LNet: Removed LNI 192.168.202.9@tcp [ 1464.760362] LNet: Removed LNI 192.168.202.9@tcp1 [ 1464.800500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1464.840699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1464.862651] LNet: Added LNI 192.168.202.9@tcp [8/256/0/180] [ 1465.527235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1465.543629] LNet: Added LNI 192.168.202.9@tcp1 [8/256/0/180] [ 1466.314819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.9@tcp [ 1466.345741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1466.403274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1466.447637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1466.491878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1466.559643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1466.602419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1466.644633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.202.9@tcp [ 1466.666340] LNetError: 68647:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.202.9@tcp to recovery [ 1466.848129] LNet: There was an unexpected network error while writing to 192.168.202.9: rc = -22 [ 1469.126535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1472.991321] LNet: There was an unexpected network error while writing to 192.168.202.9: rc = -22 [ 1472.996709] LNet: Skipped 1 previous similar message [ 1476.944359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1476.976456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1476.994111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1483.603683] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 15:19:01 (1787599141) [ 1483.757650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1483.781836] LNet: 69350:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1483.802297] LNetError: 69350:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1483.813464] LNetError: 69350:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1483.823763] LNet: Removed LNI 192.168.202.9@tcp [ 1483.839499] LNet: Removed LNI 192.168.202.9@tcp1 [ 1483.914092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1483.969436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1483.990653] LNet: Added LNI 192.168.202.9@tcp [8/256/0/180] [ 1485.171579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1485.192730] LNet: Added LNI 192.168.202.9@tcp1 [8/256/0/180] [ 1486.251723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.9@tcp [ 1486.322977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1486.344931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1486.374517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1486.404965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1486.435782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1486.494623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.202.9@tcp --health 0 [ 1486.637053] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1487.007304] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.9@tcp [ 1487.007802] LNet: There was an unexpected network error while writing to 192.168.202.9: rc = -22 [ 1487.028793] LNet: Skipped 1 previous similar message [ 1502.319070] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1502.566975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1502.662652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1502.740716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.202.9@tcp --health 500 [ 1502.872770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1509.645915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1509.725993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1509.754968] LNet: 69845:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1509.770160] LNetError: 69845:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1509.785723] LNetError: 69845:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1509.797415] LNet: Removed LNI 192.168.202.9@tcp [ 1509.811435] LNet: Removed LNI 192.168.202.9@tcp1 [ 1509.835149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1509.943720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1509.974703] LNet: Added LNI 192.168.202.9@tcp [8/256/0/180] [ 1510.931335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1510.947240] LNet: Added LNI 192.168.202.9@tcp1 [8/256/0/180] [ 1511.802536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.9@tcp [ 1511.888775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1511.982780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1512.022489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1512.057522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1512.123744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1512.175423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1512.220848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.202.9@tcp --health 0 [ 1513.952336] LNet: There was an unexpected network error while writing to 192.168.202.9: rc = -22 [ 1513.955345] LNet: Skipped 3 previous similar messages [ 1514.745598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1524.055830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1524.103317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1524.154448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1531.716485] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 15:19:49 (1787599189) [ 1532.268209] LNet: 70890:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1532.273911] LNetError: 70890:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1532.279456] LNetError: 70890:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1533.343657] LNet: Removed LNI 192.168.202.9@tcp [ 1533.355837] LNet: Removed LNI 192.168.202.9@tcp1 [ 1533.958542] Key type .llcrypt unregistered [ 1533.961280] Key type ._llcrypt unregistered [ 1540.451633] Key type ._llcrypt registered [ 1540.453195] Key type .llcrypt registered [ 1540.567924] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1540.577065] alg: No test for adler32 (adler32-zlib) [ 1541.484411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1541.504548] LNet: Added LNI 192.168.202.9@tcp [8/256/0/180] [ 1552.207160] Lustre: DEBUG MARKER: Initial discovery [ 1552.465982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.202.109@tcp [ 1555.220098] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1555.501240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1555.539848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.202.109@tcp [ 1555.560529] LNet: There was an unexpected network error while writing to 192.168.202.109: rc = -22 [ 1555.583625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1555.634653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.202.109@tcp -r 1 -m GET -l 3 [ 1555.707974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.202.109@tcp -r 1 -m GET -e local_error [ 1555.769796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.202.109@tcp -r 1 -m PUT -l 6 [ 1555.871433] LNet: 1 local NIs in recovery (showing 1): 192.168.202.9@tcp [ 1557.651442] Lustre: DEBUG MARKER: Force oleg209-client.virtnet to discover 192.168.202.109@tcp (in background) [ 1557.983455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.202.109@tcp [ 1559.702748] Lustre: DEBUG MARKER: Force oleg209-server to discover 192.168.202.9@tcp [ 1561.079029] LNet: There was an unexpected network error while writing to 192.168.202.109: rc = -22 [ 1562.734811] Lustre: DEBUG MARKER: Wait for 71659 [ 1564.889214] Lustre: DEBUG MARKER: Finished wait on 71659 [ 1564.974499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay del -a [ 1565.019362] LNet: There was an unexpected network error while writing to 192.168.202.109: rc = -22 [ 1565.055664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1565.463233] LNet: 72112:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1565.469493] LNetError: 72112:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1565.486964] LNet: Removed LNI 192.168.202.9@tcp [ 1565.804732] Key type .llcrypt unregistered [ 1565.807495] Key type ._llcrypt unregistered [ 1584.796623] Key type ._llcrypt registered [ 1584.799831] Key type .llcrypt registered [ 1586.333810] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 15:20:43 (1787599243) [ 1586.968767] Key type .llcrypt unregistered [ 1586.970628] Key type ._llcrypt unregistered [ 1592.789704] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1592.982660] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1592.993270] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1593.007763] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1595.500639] Key type ._llcrypt registered [ 1595.501920] Key type .llcrypt registered [ 1595.653695] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1595.664880] alg: No test for adler32 (adler32-zlib) [ 1596.580609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1596.639810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1596.658349] LNet: Added LNI 192.168.202.9@tcp [8/256/0/180] [ 1597.550348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1597.570822] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1607.136827] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 15:21:04 (1787599264) [ 1607.350384] LNet: 74188:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1607.363141] LNetError: 74188:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1607.375966] LNet: Removed LNI 10.1.2.3@tcp [ 1607.399593] LNet: Removed LNI 192.168.202.9@tcp [ 1608.091802] Key type .llcrypt unregistered [ 1608.113911] Key type ._llcrypt unregistered [ 1615.416572] Key type ._llcrypt registered [ 1615.419282] Key type .llcrypt registered [ 1615.547772] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1615.556955] alg: No test for adler32 (adler32-zlib) [ 1616.638697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1616.711244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1616.737919] LNet: Added LNI 192.168.202.9@tcp [8/256/0/180] [ 1617.591828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1617.624622] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1618.712729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.202.9@tcp [ 1619.743374] LNetError: Unexpected error -22 connecting to 192.168.202.9@tcp at host 192.168.202.9:988 [ 1627.264787] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 15:21:24 (1787599284) [ 1627.581957] LNet: 75326:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1627.602125] LNetError: 75326:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1627.639961] LNet: Removed LNI 10.1.2.3@tcp [ 1627.667363] LNet: Removed LNI 192.168.202.9@tcp [ 1628.864623] Key type .llcrypt unregistered [ 1628.872869] Key type ._llcrypt unregistered [ 1639.736480] Key type ._llcrypt registered [ 1639.744488] Key type .llcrypt registered [ 1639.982387] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1640.024356] alg: No test for adler32 (adler32-zlib) [ 1641.250808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1641.332804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1641.347543] LNet: Added LNI 192.168.202.9@tcp1 [8/256/0/180] [ 1641.911872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1641.941618] LNet: Added LNI 192.168.202.9@tcp2 [8/256/0/180] [ 1643.622549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.202.9@tcp1 --nid 192.168.202.9@tcp2 [ 1649.431666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.9@tcp1 192.168.202.9@tcp2 [ 1649.472753] LNetError: 76166:0:(lib-move.c:2101:lnet_handle_find_routed_path()) Unable to send message from 192.168.202.9@tcp1 to 192.168.202.9@tcp2 - Route table may be misconfigured [ 1649.559907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.9@tcp2 192.168.202.9@tcp1 [ 1657.297802] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 15:21:55 (1787599315) [ 1657.641744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1657.669829] LNet: 76739:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1657.682765] LNetError: 76739:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1658.720252] LNet: Removed LNI 192.168.202.9@tcp1 [ 1658.729234] LNetError: 76739:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1658.752578] LNet: Removed LNI 192.168.202.9@tcp2 [ 1658.782443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1658.840378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1658.868671] LNet: Added LNI 192.168.202.9@tcp [8/256/0/180] [ 1659.665468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1659.698282] LNet: Added LNI 192.168.202.9@tcp1 [8/256/0/180] [ 1660.584659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.9@tcp [ 1660.653427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.9@tcp 192.168.202.9@tcp [ 1660.708761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.9@tcp1 192.168.202.9@tcp1 [ 1660.749488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.202.9@tcp -d 192.168.202.9@tcp -e network_timeout [ 1660.795125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.202.9@tcp -d 192.168.202.9@tcp1 -e network_timeout [ 1660.847656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.202.9@tcp1 -d 192.168.202.9@tcp -e network_timeout [ 1660.902227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.202.9@tcp1 -d 192.168.202.9@tcp1 -e network_timeout [ 1660.984980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.9@tcp [ 1661.017928] LNet: There was an unexpected network error while writing to 192.168.202.9: rc = -22 [ 1661.042709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1661.171808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1661.308720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1661.858517] LNet: 1 local NIs in recovery (showing 1): 192.168.202.9@tcp [ 1667.705729] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 15:22:05 (1787599325) [ 1667.826931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1667.863090] LNet: 77599:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1667.877935] LNetError: 77599:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1667.894746] LNet: Removed LNI 192.168.202.9@tcp [ 1668.970247] LNet: Removed LNI 192.168.202.9@tcp1 [ 1669.036809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1669.137131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1669.392280] LNet: 77643:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1669.816977] Key type .llcrypt unregistered [ 1669.831902] Key type ._llcrypt unregistered [ 1681.988105] Key type ._llcrypt registered [ 1681.991179] Key type .llcrypt registered [ 1683.390585] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 15:22:21 (1787599341) [ 1683.905841] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1683.928531] alg: No test for adler32 (adler32-zlib) [ 1684.850556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1686.189685] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1687.768946] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 15:22:25 (1787599345) [ 1688.044515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1688.086094] LNet: 78742:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1688.117465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1688.229157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1688.265041] LNet: Added LNI 192.168.202.9@tcp [8/256/0/180] [ 1689.251950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1689.271340] LNet: Added LNI 192.168.202.9@tcp1 [8/256/0/180] [ 1690.100632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.9@tcp [ 1690.133933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.9@tcp1 [ 1690.173658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.9@tcp1 [ 1696.484983] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 15:22:34 (1787599354) [ 1698.068551] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg209-server [ 1700.158981] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 15:22:37 (1787599357) [ 1703.171781] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg209-server [ 1705.856863] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 15:22:42 (1787599362) [ 1707.978526] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg209-server [ 1709.561752] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 15:22:47 (1787599367) [ 1711.412111] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg209-server [ 1713.629382] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 15:22:51 (1787599371) [ 1716.098517] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg209-server [ 1718.287278] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 15:22:55 (1787599375) [ 1720.533622] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg209-server [ 1722.503609] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 15:23:00 (1787599380) [ 1724.523452] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg209-server [ 1726.699815] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 15:23:04 (1787599384) [ 1728.753418] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg209-server [ 1730.642640] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 15:23:08 (1787599388) [ 1732.590757] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg209-server [ 1734.543938] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 15:23:12 (1787599392) [ 1735.057668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1735.085439] LNet: 82620:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1735.094109] LNetError: 82620:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1735.101404] LNet: Removed LNI 192.168.202.9@tcp [ 1736.170712] LNet: Removed LNI 192.168.202.9@tcp1 [ 1736.235811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1736.331161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1736.357744] LNet: Added LNI 192.168.202.9@tcp [8/256/0/180] [ 1737.083412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1737.172566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.9@tcp [ 1737.380491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1737.396973] LNet: 82768:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1737.405811] LNetError: 82768:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1737.416249] LNetError: 82768:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1737.439419] LNet: Removed LNI 192.168.202.9@tcp [ 1737.455873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1737.514912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1737.594302] LNet: Added LNI 192.168.202.9@tcp [8/256/0/180] [ 1738.721661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.9@tcp [ 1738.976986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1739.019534] LNetError: 82913:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1740.070268] LNet: Removed LNI 192.168.202.9@tcp [ 1740.140788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1740.205214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1740.248371] LNet: Added LNI 192.168.202.9@tcp [8/256/0/180] [ 1741.064637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1741.146436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.9@tcp [ 1741.412594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1741.463812] LNet: 83061:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1741.469309] LNet: 83061:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1741.475851] LNetError: 83061:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1741.514155] LNet: Removed LNI 192.168.202.9@tcp [ 1741.544547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1741.627493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1741.658854] LNet: Added LNI 192.168.202.9@tcp [8/256/0/180] [ 1742.381995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.9@tcp [ 1742.534240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1743.590566] LNet: Removed LNI 192.168.202.9@tcp [ 1743.618906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1743.686526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1743.723262] LNet: Added LNI 192.168.202.9@tcp [8/256/0/180] [ 1744.365490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1744.430614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.9@tcp [ 1744.639970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1745.703324] LNet: Removed LNI 192.168.202.9@tcp [ 1745.750832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1745.815818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1745.838981] LNet: Added LNI 192.168.202.9@tcp [8/256/0/180] [ 1746.554657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.9@tcp [ 1746.764116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1746.777936] LNet: 83498:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1746.786101] LNet: 83498:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 1746.793429] LNetError: 83498:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1746.803190] LNetError: 83498:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1746.836776] LNet: Removed LNI 192.168.202.9@tcp [ 1746.885349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1746.945334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1746.980958] LNet: Added LNI 192.168.202.9@tcp [8/256/0/180] [ 1747.733257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1747.806319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.9@tcp [ 1747.990377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1749.036765] LNet: Removed LNI 192.168.202.9@tcp [ 1749.079512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1749.172804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1749.201362] LNet: Added LNI 192.168.202.9@tcp [8/256/0/180] [ 1749.881575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.9@tcp [ 1751.297465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1752.365536] LNet: Removed LNI 192.168.202.9@tcp [ 1752.402887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1752.498481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1752.529329] LNet: Added LNI 192.168.202.9@tcp [8/256/0/180] [ 1753.136448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1753.215535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.9@tcp [ 1753.397791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1754.468583] LNet: Removed LNI 192.168.202.9@tcp [ 1754.492902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1754.569617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1754.602915] LNet: Added LNI 192.168.202.9@tcp [8/256/0/180] [ 1755.150459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.9@tcp [ 1755.323228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1755.371318] LNet: 84093:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1755.377665] LNet: 84093:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 1755.392092] LNetError: 84093:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1755.401270] LNetError: 84093:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 1755.427776] LNet: Removed LNI 192.168.202.9@tcp [ 1755.501919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1755.595269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1755.654406] LNet: Added LNI 192.168.202.9@tcp [8/256/0/180] [ 1756.451282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1756.557386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.9@tcp [ 1758.041213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1759.078783] LNet: Removed LNI 192.168.202.9@tcp [ 1759.136970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1759.256995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1759.307350] LNet: Added LNI 192.168.202.9@tcp [8/256/0/180] [ 1760.250205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.9@tcp [ 1761.699986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1762.798270] LNet: Removed LNI 192.168.202.9@tcp [ 1762.860260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1762.933852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1762.968720] LNet: Added LNI 192.168.202.9@tcp [8/256/0/180] [ 1763.692907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1763.765050] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1763.801936] LNet: Removed LNI 192.168.202.9@tcp [ 1763.821357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1763.878591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1763.905835] LNet: Added LNI 192.168.202.9@tcp [8/256/0/180] [ 1764.740597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1764.860290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1764.892953] LNet: Removed LNI 192.168.202.9@tcp [ 1764.910413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1764.966598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 128 [ 1765.090098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1765.143309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1765.239342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 1000000000000000000000000 [ 1765.343606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1765.396874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1765.488854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer foo [ 1765.672410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1765.725151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1765.835862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 8 [ 1765.867481] LNet: Added LNI 192.168.202.9@tcp [8/256/0/180] [ 1766.559935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1766.618769] LNet: Removed LNI 192.168.202.9@tcp [ 1766.648538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1766.759404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1774.521674] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 15:23:52 (1787599432) [ 1774.777604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1774.800118] LNet: 85425:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1774.805295] LNet: 85425:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 8 previous similar messages [ 1774.830136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1774.899309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1774.939378] LNet: Added LNI 192.168.202.9@tcp231 [8/256/0/180] [ 1776.066669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1776.097809] LNetError: 85563:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1776.106995] LNetError: 85563:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 1776.136682] LNet: Removed LNI 192.168.202.9@tcp231 [ 1776.171314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1776.292675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1776.343530] LNet: Added LNI 192.168.202.9@tcp231 [8/256/0/180] [ 1776.353071] LNet: 85571:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1776.950609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1776.969921] LNet: Removed LNI 192.168.202.9@tcp231 [ 1776.983617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1777.008351] LNet: Added LNI 192.168.202.9@tcp231 [8/256/0/180] [ 1778.592413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1778.646052] LNet: Removed LNI 192.168.202.9@tcp231 [ 1778.688027] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1778.770178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1778.809118] LNet: Added LNI 192.168.202.9@tcp231 [8/256/0/0] [ 1778.817449] LNet: 85723:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1779.435270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1779.488978] LNet: Removed LNI 192.168.202.9@tcp231 [ 1779.502837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1779.564977] LNet: Added LNI 192.168.202.9@tcp231 [8/256/0/0] [ 1780.686785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1780.731234] LNet: Removed LNI 192.168.202.9@tcp231 [ 1780.765294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1780.832382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1780.861786] LNet: Added LNI 192.168.202.9@tcp231 [8/256/0/60] [ 1780.865483] LNet: 85875:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1781.164926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1781.203193] LNet: Removed LNI 192.168.202.9@tcp231 [ 1781.222209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1781.258876] LNet: Added LNI 192.168.202.9@tcp231 [8/256/0/60] [ 1782.490318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1782.540443] LNet: Removed LNI 192.168.202.9@tcp231 [ 1782.557702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1782.622225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1782.640519] LNet: Added LNI 192.168.202.9@tcp231 [8/256/0/180] [ 1783.395320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1783.442181] LNet: Removed LNI 192.168.202.9@tcp231 [ 1783.464185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1783.490759] LNet: Added LNI 192.168.202.9@tcp231 [8/256/0/180] [ 1793.014496] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 15:24:10 (1787599450) [ 1793.242790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1793.301164] LNet: Removed LNI 192.168.202.9@tcp231 [ 1793.339105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1793.410605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1793.452079] LNet: Added LNI 192.168.202.9@tcp232 [8/256/0/180] [ 1794.278381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1794.334785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1794.368147] LNet: Removed LNI 192.168.202.9@tcp232 [ 1794.391200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1794.428584] LNet: Added LNI 192.168.202.9@tcp232 [8/256/0/180] [ 1795.472161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1795.520616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1795.545527] LNet: Removed LNI 192.168.202.9@tcp232 [ 1795.563667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1795.626751] LNet: Added LNI 192.168.202.9@tcp232 [8/256/0/180] [ 1796.538593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1796.569045] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1796.598931] LNet: Removed LNI 192.168.202.9@tcp232 [ 1796.617816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1796.643069] LNet: Added LNI 192.168.202.9@tcp232 [8/256/0/180] [ 1797.508645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1797.534058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1797.561519] LNet: Removed LNI 192.168.202.9@tcp232 [ 1797.583477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1804.234575] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 15:24:22 (1787599462) [ 1805.885406] Key type .llcrypt unregistered [ 1805.892861] Key type ._llcrypt unregistered [ 1812.402431] Key type ._llcrypt registered [ 1812.408546] Key type .llcrypt registered [ 1812.702741] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1812.731845] alg: No test for adler32 (adler32-zlib) [ 1813.603470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1813.638066] LNet: Added LNI 192.168.202.9@tcp [8/256/0/180] [ 1824.142788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.109@tcp [ 1836.111871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1836.161965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1836.180132] LNet: Added LNI 192.168.202.9@tcp1 [8/256/0/180] [ 1837.664788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.9@tcp -d 192.168.202.109@tcp -m GET -e local_interrupt -r 2 [ 1837.695438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.9@tcp -d 192.168.202.109@tcp1 -m GET -e local_interrupt -r 2 [ 1837.726389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.9@tcp -d 192.168.202.9@tcp -m GET -e local_interrupt -r 2 [ 1837.760962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.9@tcp -d 192.168.202.9@tcp1 -m GET -e local_interrupt -r 2 [ 1837.795423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.9@tcp1 -d 192.168.202.109@tcp -m GET -e local_interrupt -r 2 [ 1837.826179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.9@tcp1 -d 192.168.202.109@tcp1 -m GET -e local_interrupt -r 2 [ 1837.855232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.9@tcp1 -d 192.168.202.9@tcp -m GET -e local_interrupt -r 2 [ 1837.886589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.9@tcp1 -d 192.168.202.9@tcp1 -m GET -e local_interrupt -r 2 [ 1837.918912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.9@tcp 192.168.202.109@tcp [ 1837.960623] LNet: There was an unexpected network error while writing to 192.168.202.109: rc = -22 [ 1837.963824] LNet: 1 local NIs in recovery (showing 1): 192.168.202.9@tcp [ 1838.188690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1838.203204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1838.222509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1838.247255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1838.273651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1838.302585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1838.337734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1838.364419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1838.382407] LNet: 88535:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1838.395931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1838.420501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1838.445702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1838.461839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1838.487959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1838.518597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1839.068487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.9@tcp -d 192.168.202.109@tcp -m GET -e local_dropped -r 2 [ 1839.098991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.9@tcp -d 192.168.202.109@tcp1 -m GET -e local_dropped -r 2 [ 1839.132326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.9@tcp -d 192.168.202.9@tcp -m GET -e local_dropped -r 2 [ 1839.171665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.9@tcp -d 192.168.202.9@tcp1 -m GET -e local_dropped -r 2 [ 1839.194815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.9@tcp1 -d 192.168.202.109@tcp -m GET -e local_dropped -r 2 [ 1839.217835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.9@tcp1 -d 192.168.202.109@tcp1 -m GET -e local_dropped -r 2 [ 1839.251674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.9@tcp1 -d 192.168.202.9@tcp -m GET -e local_dropped -r 2 [ 1839.327310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.9@tcp1 -d 192.168.202.9@tcp1 -m GET -e local_dropped -r 2 [ 1839.387374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.9@tcp 192.168.202.109@tcp [ 1839.408215] LNet: There was an unexpected network error while writing to 192.168.202.109: rc = -22 [ 1839.754522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1839.779686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1839.812057] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1839.838812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1839.869673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1839.902282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1839.949360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1840.020309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1840.055328] LNet: 88655:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1840.089349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1840.155165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1840.226287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1840.301797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1840.362456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1840.410987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1841.078363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.9@tcp -d 192.168.202.109@tcp -m GET -e local_aborted -r 2 [ 1841.143502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.9@tcp -d 192.168.202.109@tcp1 -m GET -e local_aborted -r 2 [ 1841.196596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.9@tcp -d 192.168.202.9@tcp -m GET -e local_aborted -r 2 [ 1841.249330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.9@tcp -d 192.168.202.9@tcp1 -m GET -e local_aborted -r 2 [ 1841.309367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.9@tcp1 -d 192.168.202.109@tcp -m GET -e local_aborted -r 2 [ 1841.357378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.9@tcp1 -d 192.168.202.109@tcp1 -m GET -e local_aborted -r 2 [ 1841.406769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.9@tcp1 -d 192.168.202.9@tcp -m GET -e local_aborted -r 2 [ 1841.460594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.9@tcp1 -d 192.168.202.9@tcp1 -m GET -e local_aborted -r 2 [ 1841.508261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.9@tcp 192.168.202.109@tcp [ 1841.552481] LNet: There was an unexpected network error while writing to 192.168.202.109: rc = -22 [ 1841.884895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1841.912231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1841.939990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1841.977745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1842.046420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1842.078473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1842.118905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1842.144396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1842.166562] LNet: 88776:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1842.189202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1842.220908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1842.245462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1842.276417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1842.304217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1842.335623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1843.062787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.9@tcp -d 192.168.202.109@tcp -m GET -e local_no_route -r 2 [ 1843.123480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.9@tcp -d 192.168.202.109@tcp1 -m GET -e local_no_route -r 2 [ 1843.170884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.9@tcp -d 192.168.202.9@tcp -m GET -e local_no_route -r 2 [ 1843.223685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.9@tcp -d 192.168.202.9@tcp1 -m GET -e local_no_route -r 2 [ 1843.263696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.9@tcp1 -d 192.168.202.109@tcp -m GET -e local_no_route -r 2 [ 1843.294655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.9@tcp1 -d 192.168.202.109@tcp1 -m GET -e local_no_route -r 2 [ 1843.326340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.9@tcp1 -d 192.168.202.9@tcp -m GET -e local_no_route -r 2 [ 1843.360936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.9@tcp1 -d 192.168.202.9@tcp1 -m GET -e local_no_route -r 2 [ 1843.394500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.9@tcp 192.168.202.109@tcp [ 1843.674367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1843.726166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1843.792482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1843.872565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1843.941598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1843.982521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1844.040577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1844.082261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1844.145505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1844.201729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1844.237648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1844.262271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1844.304666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1844.342393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1844.942479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.9@tcp -d 192.168.202.109@tcp -m GET -e local_timeout -r 2 [ 1844.968510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.9@tcp -d 192.168.202.109@tcp1 -m GET -e local_timeout -r 2 [ 1845.004054] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.9@tcp -d 192.168.202.9@tcp -m GET -e local_timeout -r 2 [ 1845.060750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.9@tcp -d 192.168.202.9@tcp1 -m GET -e local_timeout -r 2 [ 1845.098917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.9@tcp1 -d 192.168.202.109@tcp -m GET -e local_timeout -r 2 [ 1845.145641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.9@tcp1 -d 192.168.202.109@tcp1 -m GET -e local_timeout -r 2 [ 1845.177711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.9@tcp1 -d 192.168.202.9@tcp -m GET -e local_timeout -r 2 [ 1845.220564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.9@tcp1 -d 192.168.202.9@tcp1 -m GET -e local_timeout -r 2 [ 1845.259711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.9@tcp 192.168.202.109@tcp [ 1845.279679] LNet: There was an unexpected network error while writing to 192.168.202.109: rc = -22 [ 1845.286690] LNet: Skipped 1 previous similar message [ 1845.541499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1845.575641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1845.610682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1845.646298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1845.675225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1845.706287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1845.741457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1845.778742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1845.802989] LNet: 89017:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1845.815458] LNet: 89017:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1845.888659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1845.923276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1845.998210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1846.063671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1846.111540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1846.176427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1846.840445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.9@tcp -d 192.168.202.109@tcp -m GET -e remote_dropped -r 2 [ 1846.869736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.9@tcp -d 192.168.202.109@tcp1 -m GET -e remote_dropped -r 2 [ 1846.916341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.9@tcp -d 192.168.202.9@tcp -m GET -e remote_dropped -r 2 [ 1846.961552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.9@tcp -d 192.168.202.9@tcp1 -m GET -e remote_dropped -r 2 [ 1847.014765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.9@tcp1 -d 192.168.202.109@tcp -m GET -e remote_dropped -r 2 [ 1847.053578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.9@tcp1 -d 192.168.202.109@tcp1 -m GET -e remote_dropped -r 2 [ 1847.089553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.9@tcp1 -d 192.168.202.9@tcp -m GET -e remote_dropped -r 2 [ 1847.143930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.9@tcp1 -d 192.168.202.9@tcp1 -m GET -e remote_dropped -r 2 [ 1847.208288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.9@tcp 192.168.202.109@tcp [ 1847.261768] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.109@tcp [ 1847.461905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1847.496600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1847.524967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1847.561171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1847.584091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1847.618570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1847.644838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1847.673372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1847.709555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1847.752439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1847.796420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1847.837996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1847.885597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1847.915362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1853.507708] LNet: 89215:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1853.513844] LNetError: 89215:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1853.519772] LNet: Removed LNI 192.168.202.9@tcp [ 1853.530452] LNet: Removed LNI 192.168.202.9@tcp1 [ 1854.022702] Key type .llcrypt unregistered [ 1854.026706] Key type ._llcrypt unregistered [ 1870.092993] Key type ._llcrypt registered [ 1870.094916] Key type .llcrypt registered [ 1871.533747] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 15:25:29 (1787599529) [ 1872.031882] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1872.046387] alg: No test for adler32 (adler32-zlib) [ 1872.999276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1873.060275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1873.085357] LNet: Added LNI 192.168.202.9@tcp [8/256/0/180] [ 1874.263351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1874.303217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.202.9@tcp [ 1874.344124] LNetError: 90193:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.202.9@tcp to recovery [ 1874.492760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.202.9@tcp [ 1874.652403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1881.169118] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 15:25:39 (1787599539) [ 1881.562851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1881.586536] LNet: 90783:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1881.590985] LNetError: 90783:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1881.613953] LNet: Removed LNI 192.168.202.9@tcp [ 1881.647633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1881.718349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1881.738476] LNet: Added LNI 192.168.202.9@tcp [8/256/0/180] [ 1882.548449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1882.592347] LNet: Added LNI 192.168.202.9@tcp1 [8/256/0/180] [ 1883.616358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.202.9@tcp [ 1883.666931] LNetError: 91049:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.202.9@tcp to recovery [ 1883.748409] LNet: 1 local NIs in recovery (showing 1): 192.168.202.9@tcp [ 1883.757078] LNet: There was an unexpected network error while writing to 192.168.202.9: rc = -22 [ 1884.984224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.202.9@tcp [ 1893.728374] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 15:25:51 (1787599551) [ 1896.125185] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg209-server [ 1898.446814] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 15:25:55 (1787599555) [ 1900.535270] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg209-server [ 1902.511181] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 15:26:00 (1787599560) [ 1904.957604] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg209-server [ 1907.219562] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 15:26:04 (1787599564) [ 1907.347531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1907.381538] LNet: 92667:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1907.388041] LNetError: 92667:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1907.394307] LNet: Removed LNI 192.168.202.9@tcp [ 1907.407368] LNet: Removed LNI 192.168.202.9@tcp1 [ 1907.437525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1907.525816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1907.551364] LNet: Added LNI 192.168.202.9@tcp [8/256/0/180] [ 1908.481898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1908.541069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1908.578730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1908.636949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1908.674090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1908.714984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1916.171026] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 15:26:14 (1787599574) [ 1916.314839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1916.350128] LNet: 93394:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1916.359648] LNetError: 93394:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1916.373086] LNetError: 93394:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1916.435673] LNet: Removed LNI 192.168.202.9@tcp [ 1916.450284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1916.491664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1916.512878] LNet: Added LNI 192.168.202.9@tcp [8/256/0/180] [ 1924.456203] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 15:26:22 (1787599582) [ 1924.970457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1925.014657] LNet: 94100:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1925.034698] LNetError: 94100:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1925.061384] LNet: Removed LNI 192.168.202.9@tcp [ 1925.078982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1925.132260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1925.165198] LNet: Added LNI 192.168.202.9@tcp [8/256/0/180] [ 1934.929403] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 15:26:32 (1787599592) [ 1937.284354] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 1939.806388] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 15:26:37 (1787599597) [ 1941.083595] LNet: 95136:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1941.091222] LNetError: 95136:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1941.111915] LNet: Removed LNI 192.168.202.9@tcp [ 1942.423794] Key type .llcrypt unregistered [ 1942.425761] Key type ._llcrypt unregistered [ 1950.366177] Key type ._llcrypt registered [ 1950.367808] Key type .llcrypt registered [ 1950.627173] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1950.655643] alg: No test for adler32 (adler32-zlib) [ 1951.710808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1951.746459] LNet: Added LNI 192.168.202.9@tcp [8/256/0/180] [ 1959.615407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.109@tcp [ 1966.673371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.202.109@tcp [ 1967.111500] LNet: 95562:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1967.122559] LNetError: 95562:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1967.141660] LNet: Removed LNI 192.168.202.9@tcp [ 1967.855965] Key type .llcrypt unregistered [ 1967.860916] Key type ._llcrypt unregistered [ 1980.981412] Key type ._llcrypt registered [ 1980.983856] Key type .llcrypt registered [ 1982.355311] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 1985.226922] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 1987.651636] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 15:27:24 (1787599644) [ 1989.729818] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 [ 1992.320278] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 15:27:29 (1787599649) [ 1995.166790] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg209-server [ 1997.862652] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 15:27:34 (1787599654) [ 2001.020992] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg209-server [ 2004.404156] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 15:27:40 (1787599660) [ 2006.957471] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2007.051094] alg: No test for adler32 (adler32-zlib) [ 2008.290332] Lustre: Lustre: Build Version: 2.17.57_1_g9ee77b6 [ 2008.742697] LNet: Added LNI 192.168.202.9@tcp [8/256/0/180] [ 2010.471182] Key type lgssc registered [ 2012.109815] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2035.328791] Key type lgssc unregistered [ 2035.666871] LNet: 98742:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2035.677467] LNetError: 98742:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2035.702655] LNet: Removed LNI 192.168.202.9@tcp [ 2036.685720] Key type .llcrypt unregistered [ 2036.695388] Key type ._llcrypt unregistered [ 2055.449065] Key type ._llcrypt registered [ 2055.453298] Key type .llcrypt registered [ 2056.905621] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 15:28:34 (1787599714) [ 2057.617806] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2057.643978] alg: No test for adler32 (adler32-zlib) [ 2058.622921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2058.734129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2058.826398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2058.873749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2058.931390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2066.916182] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 15:28:44 (1787599724) [ 2067.258657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2067.317313] LNet: 100158:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2067.370799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2067.483113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2067.523720] LNet: Added LNI 192.168.202.9@tcp [8/256/0/180] [ 2068.887402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2068.918780] LNet: 100302:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2068.927671] LNetError: 100302:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2068.945800] LNet: Removed LNI 192.168.202.9@tcp [ 2068.978405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2069.082945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 2069.138760] LNet: Added LNI 192.168.202.9@tcp [8/256/0/180] [ 2070.225962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2070.314523] LNet: 100330:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2070.328064] LNetError: 100330:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2070.355680] LNet: Removed LNI 192.168.202.9@tcp [ 2070.385328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2070.515668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 2070.567382] LNet: Added LNI 192.168.202.9@tcp [8/256/0/180] [ 2079.271518] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 15:28:56 (1787599736) [ 2079.493451] LNet: 100915:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2079.515548] LNetError: 100915:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2079.548544] LNet: Removed LNI 192.168.202.9@tcp [ 2080.806573] Key type .llcrypt unregistered [ 2080.809463] Key type ._llcrypt unregistered [ 2088.572353] Key type ._llcrypt registered [ 2088.574922] Key type .llcrypt registered [ 2088.953092] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2088.988439] alg: No test for adler32 (adler32-zlib) [ 2090.034809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2090.187432] LNet: Added LNI 192.168.202.9@tcp [8/256/0/180] [ 2090.195192] LNet: *** cfs_fail_loc=e003, val=100*** [ 2091.244993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.9@tcp [ 2091.265898] LNet: *** cfs_fail_loc=e003, val=100*** [ 2091.274202] LNet: Skipped 2 previous similar messages [ 2091.289429] Ping reply received with mapping: [ 2091.293421] found metadata reply NI[192.168.202.9@tcp] [ 2091.298091] 09 ca a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 2091.303966] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 2091.317190] 5a 5a 5a 5a 5a 5a 5a 5a [ 2091.583918] LNet: 101322:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2091.598263] LNetError: 101322:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2092.646146] LNet: Removed LNI 192.168.202.9@tcp [ 2093.497519] Key type .llcrypt unregistered [ 2093.498924] Key type ._llcrypt unregistered [ 2108.371255] Key type ._llcrypt registered [ 2108.375193] Key type .llcrypt registered [ 2109.883671] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 15:29:27 (1787599767) [ 2110.616349] Key type .llcrypt unregistered [ 2110.618283] Key type ._llcrypt unregistered [ 2119.445297] Key type ._llcrypt registered [ 2119.447982] Key type .llcrypt registered [ 2119.700591] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2119.732438] alg: No test for adler32 (adler32-zlib) [ 2120.913978] LNetError: 102381:0:(api-ni.c:2850:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 2121.540344] Key type .llcrypt unregistered [ 2121.544244] Key type ._llcrypt unregistered [ 2128.674074] Key type ._llcrypt registered [ 2128.675544] Key type .llcrypt registered [ 2129.977591] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 15:29:47 (1787599787) [ 2131.199291] Key type .llcrypt unregistered [ 2131.207462] Key type ._llcrypt unregistered [ 2139.551093] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2139.560404] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2142.593877] Key type ._llcrypt registered [ 2142.602318] Key type .llcrypt registered [ 2142.970174] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2142.984970] alg: No test for adler32 (adler32-zlib) [ 2143.916461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2143.980672] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2155.665972] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 15:30:12 (1787599812) [ 2155.899931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2155.948981] LNet: 104179:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2155.957288] LNetError: 104179:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2155.969661] LNet: Removed LNI 10.1.2.3@tcp [ 2156.027818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2156.840743] Key type .llcrypt unregistered [ 2156.845675] Key type ._llcrypt unregistered [ 2165.393933] Key type ._llcrypt registered [ 2165.414939] Key type .llcrypt registered [ 2165.837289] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2165.878297] alg: No test for adler32 (adler32-zlib) [ 2167.025222] LNet: Added LNI 192.168.202.9@tcp [8/256/0/180] [ 2167.152237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.9@tcp [ 2167.521378] LNet: 104473:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2167.574661] LNetError: 104473:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2167.623094] LNet: Removed LNI 192.168.202.9@tcp [ 2169.035522] Key type .llcrypt unregistered [ 2169.041769] Key type ._llcrypt unregistered [ 2188.654551] Key type ._llcrypt registered [ 2188.656575] Key type .llcrypt registered [ 2190.392856] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 15:30:47 (1787599847) [ 2191.060307] Key type .llcrypt unregistered [ 2191.061752] Key type ._llcrypt unregistered [ 2200.186365] Key type ._llcrypt registered [ 2200.191225] Key type .llcrypt registered [ 2200.429319] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2200.459395] alg: No test for adler32 (adler32-zlib) [ 2208.476046] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 15:31:06 (1787599866) [ 2208.817842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2208.880034] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2209.005797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2209.035467] LNet: Added LNI 192.168.202.9@tcp [8/256/0/180] [ 2210.000558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2210.064598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 2210.115778] LNet: Removed LNI 192.168.202.9@tcp [ 2210.378617] LNet: 106230:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2210.808394] Key type .llcrypt unregistered [ 2210.810490] Key type ._llcrypt unregistered [ 2225.343224] Key type ._llcrypt registered [ 2225.347155] Key type .llcrypt registered [ 2227.108972] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 15:31:24 (1787599884) [ 2227.858394] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2227.888303] alg: No test for adler32 (adler32-zlib) [ 2228.893181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2228.978810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2229.022134] LNet: Added LNI 192.168.202.9@tcp [8/256/0/180] [ 2229.714577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.9@tcp [ 2229.748250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.202.9@tcp [ 2229.772428] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 2229.777297] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 2229.780565] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2229, 0, 0 [ 2229.785763] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 2229.788454] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 2229.793551] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 2229.798895] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2229, 0, 0 [ 2229.804231] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 2229.807728] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2229.814270] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 2229.817778] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2229, 0, 2279 [ 2229.823805] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 2229.829842] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2229.832925] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 2229.836284] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2229, 0, 2279 [ 2229.839946] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 2237.618907] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 15:31:35 (1787599895) [ 2238.342536] LNet: 107771:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2238.358732] LNetError: 107771:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2238.379792] LNet: Removed LNI 192.168.202.9@tcp [ 2239.204537] Key type .llcrypt unregistered [ 2239.211451] Key type ._llcrypt unregistered [ 2247.601751] Key type ._llcrypt registered [ 2247.604488] Key type .llcrypt registered [ 2247.978316] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2248.007586] alg: No test for adler32 (adler32-zlib) [ 2249.006941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2249.045131] LNet: Added LNI 192.168.202.9@tcp [8/256/0/180] [ 2258.178556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.109@tcp [ 2270.168140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2270.235047] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2270.265200] LNet: Added LNI 192.168.202.9@tcp1 [8/256/0/180] [ 2271.445660] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2271.622705] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2271.625714] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2271.639541] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2274.135091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 2274.172528] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 2287.744653] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 15:32:25 (1787599945) [ 2287.891666] LNet: 109270:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2287.921416] LNetError: 109270:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2287.927434] LNet: Removed LNI 192.168.202.9@tcp [ 2288.992107] LNet: Removed LNI 192.168.202.9@tcp1 [ 2288.996163] LNetError: 109270:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2289.015223] LNetError: 109270:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2289.081481] LNet: Removed LNI 10.1.2.3@tcp99 [ 2290.168802] Key type .llcrypt unregistered [ 2290.177203] Key type ._llcrypt unregistered [ 2297.893415] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2297.903652] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2300.624890] Key type ._llcrypt registered [ 2300.628470] Key type .llcrypt registered [ 2300.941515] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2300.974195] alg: No test for adler32 (adler32-zlib) [ 2301.997554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2302.060589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2302.084730] LNet: Added LNI 192.168.202.9@tcp [8/256/0/180] [ 2303.187546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2303.222324] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2305.822910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 2305.884362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2305.916064] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 2305.940717] LNetError: 110025:0:(peer.c:2362:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 2305.986255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2306.032859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.9@tcp [ 2306.726715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2306.859347] LNet: 110045:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2306.867292] LNetError: 110045:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2306.875724] LNet: Removed LNI 10.1.2.3@tcp [ 2306.896184] LNet: Removed LNI 192.168.202.9@tcp [ 2307.578614] Key type .llcrypt unregistered [ 2307.580171] Key type ._llcrypt unregistered [ 2322.561715] Key type ._llcrypt registered [ 2322.563614] Key type .llcrypt registered [ 2323.918662] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 15:33:01 (1787599981) [ 2324.832367] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2324.895954] alg: No test for adler32 (adler32-zlib) [ 2325.900767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2325.945491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2325.965875] LNet: Added LNI 192.168.202.9@tcp [8/256/0/180] [ 2327.101964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.9@tcp [ 2327.224262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg209-client@tcp [ 2336.676337] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 15:33:13 (1787599993) [ 2336.906493] LNet: 111576:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2336.925429] LNetError: 111576:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2336.963394] LNet: Removed LNI 192.168.202.9@tcp [ 2337.765481] Key type .llcrypt unregistered [ 2337.767192] Key type ._llcrypt unregistered [ 2348.653946] Key type ._llcrypt registered [ 2348.657043] Key type .llcrypt registered [ 2348.915374] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2348.930378] alg: No test for adler32 (adler32-zlib) [ 2349.875498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 2349.907000] LNet: Added LNI 192.168.202.9@tcp [8/256/0/180] [ 2349.953715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 2359.216643] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 15:33:36 (1787600016) [ 2361.727501] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 2364.535979] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 15:33:41 (1787600021) [ 2366.528877] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 2368.616636] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 15:33:46 (1787600026) [ 2369.015568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2369.066916] LNet: 113007:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2369.081756] LNetError: 113007:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2369.117557] LNet: Removed LNI 192.168.202.9@tcp [ 2369.198145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2369.338822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2369.376510] LNet: Added LNI 192.168.202.9@tcp [8/256/0/180] [ 2370.397226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2370.471644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2378.473912] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 15:33:56 (1787600036) [ 2378.590879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2378.612482] LNet: 113711:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2378.619589] LNetError: 113711:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2378.657900] LNet: Removed LNI 192.168.202.9@tcp [ 2378.716996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2378.803769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 2378.845709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2379.723547] Key type .llcrypt unregistered [ 2379.726190] Key type ._llcrypt unregistered [ 2393.172183] Key type ._llcrypt registered [ 2393.174040] Key type .llcrypt registered [ 2395.246745] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 15:34:12 (1787600052) [ 2395.776879] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2395.803581] alg: No test for adler32 (adler32-zlib) [ 2396.786346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2396.851755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2396.877244] LNet: Added LNI 192.168.202.9@tcp [8/256/0/180] [ 2397.872912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2397.955451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.9@tcp [ 2406.356212] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 15:34:24 (1787600064) [ 2406.680130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2406.721601] LNet: 115277:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2406.728519] LNetError: 115277:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2406.751865] LNet: Removed LNI 192.168.202.9@tcp [ 2406.807552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2406.890462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 2406.944463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 2415.374936] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 15:34:33 (1787600073) [ 2415.721916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2415.771922] LNet: 115857:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2415.806374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2415.923969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2416.423136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2416.485724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2416.574446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 2417.205652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2417.246662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2417.341768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2417.372389] LNet: 115890:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2417.405617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2417.491139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2418.180290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2418.198195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2418.247049] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 2418.481443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2418.502818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2427.502326] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 15:34:44 (1787600084) [ 2427.633572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2427.667095] LNet: 116488:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2427.717273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2427.777519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2427.800352] LNet: Added LNI 192.168.202.9@tcp [8/256/0/180] [ 2428.546528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2428.581907] LNet: Added LNI 192.168.202.9@tcp2 [8/256/0/180] [ 2429.912520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2430.073134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.9@tcp [ 2430.490497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2430.537443] LNetError: 116771:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2430.550505] LNet: Removed LNI 192.168.202.9@tcp [ 2430.571760] LNet: Removed LNI 192.168.202.9@tcp2 [ 2430.626328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2430.717969] LNet: Added LNI 192.168.202.9@tcp [8/256/0/180] [ 2430.726987] LNet: Added LNI 192.168.202.9@tcp2 [8/256/0/180] [ 2430.733817] LNet: 116777:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2431.553698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.9@tcp [ 2431.776794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2431.805291] LNet: 116815:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2431.809166] LNet: 116815:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 2431.819732] LNetError: 116815:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2431.830333] LNetError: 116815:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2431.838314] LNet: Removed LNI 192.168.202.9@tcp [ 2431.850570] LNet: Removed LNI 192.168.202.9@tcp2 [ 2431.890593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2431.950562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2431.971381] LNet: Added LNI 192.168.202.9@tcp [8/256/0/180] [ 2432.906791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2432.940610] LNet: Added LNI 192.168.202.9@tcp2 [8/256/0/180] [ 2434.128441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.9@tcp [ 2434.161632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2434.205661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.9@tcp [ 2434.420743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2434.444234] LNetError: 117099:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2434.449639] LNetError: 117099:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2434.453222] LNet: Removed LNI 192.168.202.9@tcp [ 2435.493913] LNet: Removed LNI 192.168.202.9@tcp2 [ 2435.541865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2435.632052] LNet: Added LNI 192.168.202.9@tcp [8/256/0/180] [ 2435.636713] LNet: Added LNI 192.168.202.9@tcp2 [8/256/0/180] [ 2435.651524] LNet: 117105:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2436.580433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.9@tcp [ 2436.870120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2436.916655] LNet: Removed LNI 192.168.202.9@tcp [ 2438.009332] LNet: Removed LNI 192.168.202.9@tcp2 [ 2438.085506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2438.160824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2438.200400] LNet: Added LNI 192.168.202.9@tcp [8/256/0/180] [ 2439.018899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2439.036445] LNet: Added LNI 192.168.202.9@tcp2 [8/256/0/180] [ 2439.830300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2439.878531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.9@tcp [ 2440.087454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2440.108530] LNet: 117422:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2440.112632] LNet: 117422:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 2440.116433] LNetError: 117422:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2440.119343] LNetError: 117422:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 2440.125296] LNet: Removed LNI 192.168.202.9@tcp [ 2440.135658] LNet: Removed LNI 192.168.202.9@tcp2 [ 2440.151317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2440.202497] LNet: Added LNI 192.168.202.9@tcp [8/256/0/180] [ 2440.211408] LNet: Added LNI 192.168.202.9@tcp2 [8/256/0/180] [ 2440.215639] LNet: 117428:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2440.933504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.9@tcp [ 2441.236462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2442.271223] LNet: Removed LNI 192.168.202.9@tcp [ 2442.288223] LNet: Removed LNI 192.168.202.9@tcp2 [ 2442.339744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2442.441837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2442.466147] LNet: Added LNI 192.168.202.9@tcp [8/256/0/180] [ 2443.375878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2443.414154] LNet: Added LNI 192.168.202.9@tcp2 [8/256/0/180] [ 2444.796316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.9@tcp [ 2444.856036] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2444.944480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.9@tcp [ 2445.193851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2446.242067] LNet: Removed LNI 192.168.202.9@tcp [ 2447.279179] LNet: Removed LNI 192.168.202.9@tcp2 [ 2447.327297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2447.428487] LNet: Added LNI 192.168.202.9@tcp [8/256/0/180] [ 2447.435573] LNet: Added LNI 192.168.202.9@tcp2 [8/256/0/180] [ 2447.444862] LNet: 117756:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2448.454964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.9@tcp [ 2448.769814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2448.802260] LNetError: 117793:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2448.808050] LNetError: 117793:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 2448.812765] LNet: Removed LNI 192.168.202.9@tcp [ 2449.830428] LNet: Removed LNI 192.168.202.9@tcp2 [ 2449.889373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2450.002723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2450.040622] LNet: Added LNI 192.168.202.9@tcp [8/256/0/180] [ 2451.160890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2451.199873] LNet: Added LNI 192.168.202.9@tcp2 [8/256/0/180] [ 2452.522666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2452.576910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2452.659475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.9@tcp [ 2453.050673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2454.111203] LNet: Removed LNI 192.168.202.9@tcp [ 2454.129560] LNet: Removed LNI 192.168.202.9@tcp2 [ 2454.203411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2454.342910] LNet: Added LNI 192.168.202.9@tcp [8/256/0/180] [ 2454.357278] LNet: Added LNI 192.168.202.9@tcp2 [8/256/0/180] [ 2454.364311] LNet: 118096:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2455.162171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.9@tcp [ 2455.689902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2456.799196] LNet: Removed LNI 192.168.202.9@tcp [ 2456.831076] LNet: Removed LNI 192.168.202.9@tcp2 [ 2456.859385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2456.949578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2456.967559] LNet: Added LNI 192.168.202.9@tcp [8/256/0/180] [ 2457.767500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2457.789496] LNet: Added LNI 192.168.202.9@tcp2 [8/256/0/180] [ 2458.634897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.9@tcp [ 2458.700237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2458.751627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2458.824892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.9@tcp [ 2459.240602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2459.263391] LNet: 118450:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2459.269108] LNet: 118450:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 5 previous similar messages [ 2460.319215] LNet: Removed LNI 192.168.202.9@tcp [ 2461.362707] LNet: Removed LNI 192.168.202.9@tcp2 [ 2461.410360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2461.527445] LNet: Added LNI 192.168.202.9@tcp [8/256/0/180] [ 2461.544305] LNet: Added LNI 192.168.202.9@tcp2 [8/256/0/180] [ 2462.441589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.9@tcp [ 2470.992725] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 2473.000913] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 15:35:30 (1787600130) [ 2473.250133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2473.297705] LNetError: 119216:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2473.310366] LNetError: 119216:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 2473.338655] LNet: Removed LNI 192.168.202.9@tcp [ 2474.408687] LNet: Removed LNI 192.168.202.9@tcp2 [ 2474.455693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2474.848249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2474.876878] LNet: Added LNI 192.168.202.9@tcp [8/256/0/180] [ 2475.660651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2475.686900] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2476.547640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2476.580385] LNet: Removed LNI 10.1.2.3@tcp [ 2476.592690] LNet: Removed LNI 192.168.202.9@tcp [ 2476.618509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2476.682683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 2476.716808] LNet: Added LNI 192.168.202.9@tcp [8/256/0/180] [ 2476.721757] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2477.151294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 2477.181124] LNet: Removed LNI 192.168.202.9@tcp [ 2477.194231] LNet: Removed LNI 10.1.2.3@tcp [ 2485.200794] Lustre: DEBUG MARKER: == sanity-lnet test 475: per-operation latency stats populate and reset ========================================================== 15:35:42 (1787600142) [ 2485.380867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2485.452273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2485.546983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2485.591099] LNet: Added LNI 192.168.202.9@tcp [8/256/0/180] [ 2486.689290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2486.752827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.9@tcp [ 2486.820589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.9@tcp [ 2486.862985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.9@tcp [ 2486.930244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.9@tcp [ 2486.989312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.9@tcp [ 2487.026001] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.9@tcp [ 2487.071861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.9@tcp [ 2487.155614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.9@tcp [ 2487.218985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.9@tcp [ 2487.286091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.9@tcp [ 2487.596898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2495.321547] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 15:35:53 (1787600153) [ 2495.621142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2495.664476] LNet: 120832:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2495.670972] LNet: 120832:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 2496.741730] LNet: Removed LNI 192.168.202.9@tcp [ 2496.792020] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2496.903626] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2497.047178] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2497.061614] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2497.070726] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2499.486320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2499.504565] LNet: Added LNI 192.168.202.9@tcp [8/256/0/180] [ 2500.052418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2500.068649] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2502.056950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.9@tcp [ 2502.122924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 2502.191392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.202.9@tcp -m PUT -e local_timeout -r 1 [ 2502.290937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 2502.403721] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2502.408764] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2502.711539] LNetError: 120893:0:(lib-move.c:4801:lnet_parse()) 192.168.202.9@tcp, src 192.168.202.9@tcp: Dropping PUT (error -108 looking up sender) [ 2503.735844] LNet: Removed LNI 10.1.2.3@tcp [ 2504.807539] LNet: Removed LNI 192.168.202.9@tcp [ 2505.547600] Key type .llcrypt unregistered [ 2505.553865] Key type ._llcrypt unregistered [ 2521.340932] Key type ._llcrypt registered [ 2521.344324] Key type .llcrypt registered [ 2522.476742] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 15:36:20 (1787600180) [ 2522.902372] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2522.916505] alg: No test for adler32 (adler32-zlib) [ 2523.958894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2524.429913] LNet: 122207:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2525.208390] Key type .llcrypt unregistered [ 2525.210497] Key type ._llcrypt unregistered [ 2532.587498] Key type ._llcrypt registered [ 2532.591853] Key type .llcrypt registered [ 2532.903117] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2532.921669] alg: No test for adler32 (adler32-zlib) [ 2533.992630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2534.042320] LNet: Added LNI 192.168.202.9@tcp [8/256/0/180] [ 2542.728548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.109@tcp [ 2543.019755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 2543.841546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.202.109@tcp [ 2544.194865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2544.246437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2544.289191] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.109@tcp [ 2544.309231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2544.361762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2544.395857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2544.444384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2544.499314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2544.569824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2544.615034] LNet: 122661:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2544.656415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2544.724270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2544.769131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2544.821777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2544.853738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2544.891241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2545.812495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2546.073605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2546.104571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2546.156988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2546.204597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2546.253361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2546.301949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2546.356552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2546.399539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2546.425823] LNet: 122778:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2546.456386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2546.501166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2546.540534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2546.582239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2546.617963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2546.652919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2547.338460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.202.109@tcp [ 2547.705388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2547.738423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2547.766685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2547.794676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2547.826341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2547.851928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2547.883152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2547.916952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2547.931575] LNet: 122896:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2547.948374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2547.976035] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2548.008975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2548.037101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2548.057913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2548.080161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2548.652532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2548.930788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2548.964691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2549.015540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2549.042657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2549.072989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2549.098311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2549.118757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2549.144817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2549.193712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2549.244601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2549.312327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2549.384270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2549.433368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2549.490107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2549.865723] LNet: 123063:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2549.871301] LNetError: 123063:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2549.885319] LNet: Removed LNI 192.168.202.9@tcp [ 2550.259812] Key type .llcrypt unregistered [ 2550.262514] Key type ._llcrypt unregistered [ 2567.196834] Key type ._llcrypt registered [ 2567.197969] Key type .llcrypt registered [ 2568.551262] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 15:37:06 (1787600226) [ 2569.332832] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2569.359237] alg: No test for adler32 (adler32-zlib) [ 2570.303363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2570.831668] LNet: 123950:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2571.700383] Key type .llcrypt unregistered [ 2571.706032] Key type ._llcrypt unregistered [ 2580.124706] Key type ._llcrypt registered [ 2580.137741] Key type .llcrypt registered [ 2580.341857] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2580.363732] alg: No test for adler32 (adler32-zlib) [ 2581.553601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2581.594855] LNet: Added LNI 192.168.202.9@tcp [8/256/0/180] [ 2591.985722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.109@tcp [ 2603.310058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2603.358507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2603.381794] LNet: Added LNI 192.168.202.9@tcp1 [8/256/0/180] [ 2604.892828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 2606.082775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.202.109@tcp [ 2606.175127] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.109@tcp [ 2606.374582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2606.422419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2606.462893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2606.516862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2606.586940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2606.649301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2606.743289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2606.808887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2606.839060] LNet: 124607:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2606.867774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2606.910987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2606.952872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2607.027771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2607.069911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2607.102335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2608.176546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2608.572989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2608.612896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2608.648311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2608.693173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2608.731586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2608.779698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2608.838572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2608.874591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2608.895630] LNet: 124725:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2608.929582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2608.971127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2609.002162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2609.026220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2609.052741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2609.081562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2609.746881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.202.109@tcp [ 2610.129588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2610.206526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2610.259150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2610.313217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2610.351227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2610.416404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2610.461402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2610.509356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2610.529520] LNet: 124842:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2610.547360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2610.600788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2610.648077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2610.686744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2610.718390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2610.749968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2611.450379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2611.745685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2611.786175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2611.814678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2611.849819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2611.883410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2611.911571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2611.936418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2611.960565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2612.006817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2612.065646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2612.120032] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2612.177904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2612.243783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2612.379362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2617.447237] LNet: 125044:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2617.451306] LNetError: 125044:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2617.457679] LNet: Removed LNI 192.168.202.9@tcp [ 2617.467867] LNet: Removed LNI 192.168.202.9@tcp1 [ 2617.785743] Key type .llcrypt unregistered [ 2617.787715] Key type ._llcrypt unregistered [ 2638.023586] Key type ._llcrypt registered [ 2638.028382] Key type .llcrypt registered [ 2639.738155] Lustre: DEBUG MARKER: == sanity-lnet test 525: keep reachable peer on local net delete ========================================================== 15:38:17 (1787600297) [ 2640.255159] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2640.276605] alg: No test for adler32 (adler32-zlib) [ 2641.265115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2641.318703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2641.348553] LNet: Added LNI 192.168.202.9@tcp [8/256/0/180] [ 2642.018964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2642.037744] LNet: Added LNI 192.168.202.9@tcp2 [8/256/0/180] [ 2643.033227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.9@tcp [ 2643.104386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp [ 2644.127276] LNet: Removed LNI 192.168.202.9@tcp [ 2651.278444] Lustre: DEBUG MARKER: == sanity-lnet test 550: DD-off: keep route up when gw reachable via another local NI ========================================================== 15:38:29 (1787600309) [ 2653.372211] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_550 Need at least 2 local interfaces for multi-rail [ 2655.202339] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 15:38:33 (1787600313) [ 2663.496179] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 15:38:41 (1787600321) [ 2671.587782] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 15:38:49 (1787600329) [ 2678.960321] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 15:38:56 (1787600336) [ 2687.517594] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 15:39:05 (1787600345) [ 2694.654196] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 15:39:12 (1787600352) [ 2702.939576] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 15:39:20 (1787600360) [ 2710.984633] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 15:39:28 (1787600368) [ 2719.125174] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 15:39:37 (1787600377) [ 2726.928298] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 15:39:44 (1787600384) [ 2735.475869] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 15:39:53 (1787600393) [ 2743.125547] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 15:40:01 (1787600401) [ 2750.869741] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 15:40:08 (1787600408) [ 2758.419777] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 15:40:16 (1787600416) [ 2766.913451] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 15:40:24 (1787600424) [ 2774.941767] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 15:40:33 (1787600433) [ 2783.317603] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 15:40:40 (1787600440) [ 2794.422811] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 15:40:51 (1787600451) [ 2794.517747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2794.553946] LNet: 136729:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2794.560157] LNetError: 136729:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2794.579086] LNet: Removed LNI 192.168.202.9@tcp2 [ 2794.606981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2794.827515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2794.896414] LNet: Added LNI 192.168.202.9@tcp [8/256/0/180] [ 2796.645387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2796.671992] LNet: 136881:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2796.676735] LNetError: 136881:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2796.698891] LNet: Removed LNI 192.168.202.9@tcp [ 2796.718652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2796.790322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2796.835626] LNet: Added LNI 192.168.202.9@tcp [8/256/0/180] [ 2805.490674] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 15:41:03 (1787600463) [ 2815.318658] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 15:41:12 (1787600472) [ 2827.175679] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 15:41:24 (1787600484) [ 2834.444776] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 15:41:32 (1787600492) [ 2843.686400] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 15:41:41 (1787600501) [ 2850.947429] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 15:41:49 (1787600509) [ 2859.086378] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 15:41:56 (1787600516) [ 2866.784714] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 15:42:04 (1787600524) [ 2874.755433] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 15:42:12 (1787600532) [ 2883.681680] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 15:42:21 (1787600541) [ 2892.657378] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 15:42:30 (1787600550) [ 2903.755113] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 15:42:40 (1787600560) [ 2915.130342] Lustre: DEBUG MARKER: == sanity-lnet test 630: Verify lnetctl peer show -v 4 doesn't crash on a multi-skb dump (LU-20587) ========================================================== 15:42:52 (1787600572) [ 2915.458061] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2915.492720] LNet: 144350:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2915.503121] LNetError: 144350:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2915.523274] LNet: Removed LNI 192.168.202.9@tcp [ 2915.565050] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2915.653628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2915.697291] LNet: Added LNI 192.168.202.9@tcp [8/256/0/180] [ 2941.739345] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 2657 sec ========== 15:43:19 (1787600599) [ 2942.329196] LNet: 145952:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2942.345191] LNetError: 145952:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2942.367212] LNet: Removed LNI 192.168.202.9@tcp [ 2943.249924] Key type .llcrypt unregistered [ 2943.252105] Key type ._llcrypt unregistered [ 2957.374529] Key type ._llcrypt registered [ 2957.388152] Key type .llcrypt registered [ 2957.799964] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2957.816503] alg: No test for adler32 (adler32-zlib) [ 2958.977803] Lustre: Lustre: Build Version: 2.17.57_1_g9ee77b6 [ 2959.363268] LNet: Added LNI 192.168.202.9@tcp [8/256/0/180] [ 2961.087327] Key type lgssc registered [ 2963.152402] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3039.800979] Lustre: Mounted lustre-client - version 2.17.57_1_g9ee77b6 [ 3044.841212] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3053.459901] Lustre: Unmounted lustre-client [ 3082.728996] Key type lgssc unregistered [ 3083.027577] LNet: 148912:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3083.036722] LNetError: 148912:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3083.077250] LNet: Removed LNI 192.168.202.9@tcp [ 3083.877172] Key type .llcrypt unregistered [ 3083.878819] Key type ._llcrypt unregistered