[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 417062559 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2421 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE22BD 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 00227D (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2331 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE23C1 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE23F9 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe22bd-0xbffe2330] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22bc] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2331-0xbffe23c0] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe23c1-0xbffe23f8] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe23f9-0xbffe2420] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001010] APIC: Switch to symmetric I/O mode setup [ 0.002353] x2apic enabled [ 0.003007] Switched APIC routing to physical x2apic. [ 0.004014] kvm-guest: setup PV IPIs [ 0.007335] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008028] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009010] pid_max: default: 32768 minimum: 301 [ 0.011033] LSM: Security Framework initializing [ 0.012007] Yama: becoming mindful. [ 0.013041] SELinux: Initializing. [ 0.014047] *** VALIDATE selinux *** [ 0.020652] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.024249] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.025125] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027046] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028090] *** VALIDATE tmpfs *** [ 0.029355] *** VALIDATE proc *** [ 0.030177] *** VALIDATE cgroup *** [ 0.031005] *** VALIDATE cgroup2 *** [ 0.032184] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.034059] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.035008] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.036022] Spectre V2 : User space: Vulnerable [ 0.037005] Speculative Store Bypass: Vulnerable [ 0.039631] debug: unmapping init [mem 0xffffffffaac59000-0xffffffffaac60fff] [ 0.042181] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.043561] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.044020] ... version: 2 [ 0.045009] ... bit width: 48 [ 0.045802] ... generic registers: 4 [ 0.046010] ... value mask: 0000ffffffffffff [ 0.047013] ... max period: 00007fffffffffff [ 0.048009] ... fixed-purpose events: 3 [ 0.048922] ... event mask: 000000070000000f [ 0.049262] rcu: Hierarchical SRCU implementation. [ 0.051204] smp: Bringing up secondary CPUs ... [ 0.052498] x86: Booting SMP configuration: [ 0.053029] .... node #0, CPUs: #1 #2 #3 [ 0.056171] smp: Brought up 1 node, 4 CPUs [ 0.057776] smpboot: Max logical packages: 1 [ 0.058013] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.141885] node 0 deferred pages initialised in 81ms [ 0.145093] devtmpfs: initialized [ 0.146315] x86/mm: Memory block size: 128MB [ 0.149785] gcov: version magic: 0x41383552 [ 0.153208] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.154079] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.156376] pinctrl core: initialized pinctrl subsystem [ 0.158202] [ 0.158696] ************************************************************* [ 0.161014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.163012] ** ** [ 0.165014] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.168015] ** ** [ 0.170011] ** This means that this kernel is built to expose internal ** [ 0.172013] ** IOMMU data structures, which may compromise security on ** [ 0.175014] ** your system. ** [ 0.177081] ** ** [ 0.179012] ** If you see this message and you are not debugging the ** [ 0.181012] ** kernel, report this immediately to your vendor! ** [ 0.183013] ** ** [ 0.185011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.187012] ************************************************************* [ 0.190708] NET: Registered protocol family 16 [ 0.192524] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.194060] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.197062] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.200104] cpuidle: using governor menu [ 0.201880] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.204444] PCI: Using configuration type 1 for base access [ 0.207128] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.218064] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.220015] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.225029] cryptd: max_cpu_qlen set to 1000 [ 0.227261] ACPI: Added _OSI(Module Device) [ 0.229012] ACPI: Added _OSI(Processor Device) [ 0.231012] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.232011] ACPI: Added _OSI(Processor Aggregator Device) [ 0.237000] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.241563] ACPI: Interpreter enabled [ 0.243059] ACPI: PM: (supports S0 S3 S4 S5) [ 0.244012] ACPI: Using IOAPIC for interrupt routing [ 0.246092] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.249364] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.259327] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.261057] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.264023] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.268096] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.274834] acpiphp: Slot [2] registered [ 0.276122] acpiphp: Slot [5] registered [ 0.278124] acpiphp: Slot [6] registered [ 0.280150] acpiphp: Slot [3] registered [ 0.281418] acpiphp: Slot [4] registered [ 0.284130] acpiphp: Slot [7] registered [ 0.285164] acpiphp: Slot [8] registered [ 0.288171] acpiphp: Slot [9] registered [ 0.290142] acpiphp: Slot [10] registered [ 0.292122] acpiphp: Slot [11] registered [ 0.293135] acpiphp: Slot [12] registered [ 0.295120] acpiphp: Slot [13] registered [ 0.296107] acpiphp: Slot [14] registered [ 0.298131] acpiphp: Slot [15] registered [ 0.299105] acpiphp: Slot [16] registered [ 0.301113] acpiphp: Slot [17] registered [ 0.302148] acpiphp: Slot [18] registered [ 0.304147] acpiphp: Slot [19] registered [ 0.306109] acpiphp: Slot [20] registered [ 0.308069] acpiphp: Slot [21] registered [ 0.309115] acpiphp: Slot [22] registered [ 0.311130] acpiphp: Slot [23] registered [ 0.312081] acpiphp: Slot [24] registered [ 0.314104] acpiphp: Slot [25] registered [ 0.316167] acpiphp: Slot [26] registered [ 0.319155] acpiphp: Slot [27] registered [ 0.321107] acpiphp: Slot [28] registered [ 0.323149] acpiphp: Slot [29] registered [ 0.325128] acpiphp: Slot [30] registered [ 0.327119] acpiphp: Slot [31] registered [ 0.329105] PCI host bridge to bus 0000:00 [ 0.331026] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.333028] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.335023] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.338033] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.341039] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.344028] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.346248] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.350265] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.354117] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.364017] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.367000] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.370023] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.372019] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.375016] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.377608] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.380767] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.382043] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.384652] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.388013] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.394836] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.398015] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.402529] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.406013] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.411014] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.422015] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.430212] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.434012] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.439024] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.453016] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.462142] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.464492] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.466486] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.468453] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.470356] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.475089] iommu: Default domain type: Passthrough [ 0.477517] SCSI subsystem initialized [ 0.478169] ACPI: bus type USB registered [ 0.480095] usbcore: registered new interface driver usbfs [ 0.482074] usbcore: registered new interface driver hub [ 0.483079] usbcore: registered new device driver usb [ 0.485164] pps_core: LinuxPPS API ver. 1 registered [ 0.486009] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.489057] PTP clock support registered [ 0.491034] EDAC MC: Ver: 3.0.0 [ 0.492118] PCI: Using ACPI for IRQ routing [ 0.493676] NetLabel: Initializing [ 0.495011] NetLabel: domain hash size = 128 [ 0.496007] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.497070] NetLabel: unlabeled traffic allowed by default [ 0.500127] vgaarb: loaded [ 0.502226] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.503009] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.508911] clocksource: Switched to clocksource kvm-clock [ 0.621470] VFS: Disk quotas dquot_6.6.0 [ 0.622803] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.624653] *** VALIDATE ramfs *** [ 0.625783] *** VALIDATE hugetlbfs *** [ 0.627145] pnp: PnP ACPI init [ 0.629881] pnp: PnP ACPI: found 6 devices [ 0.644284] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.646869] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.648569] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.650401] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.652291] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.653946] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.656157] NET: Registered protocol family 2 [ 0.658192] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.662659] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.665830] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.669718] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.672518] TCP: Hash tables configured (established 65536 bind 65536) [ 0.675498] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.677667] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.679759] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.682242] NET: Registered protocol family 1 [ 0.684221] RPC: Registered named UNIX socket transport module. [ 0.685763] RPC: Registered udp transport module. [ 0.687624] RPC: Registered tcp transport module. [ 0.688952] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.691106] NET: Registered protocol family 44 [ 0.692606] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.694568] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.696599] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.699831] PCI: CLS 0 bytes, default 64 [ 0.701382] Unpacking initramfs... [ 2.085486] debug: unmapping init [mem 0xffff9cc23cc64000-0xffff9cc23ffcffff] [ 2.091817] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.094230] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.098191] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.625769] Initialise system trusted keyrings [ 2.628375] Key type blacklist registered [ 2.632288] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.641801] zbud: loaded [ 2.645077] *** VALIDATE nfs *** [ 2.646102] *** VALIDATE nfs4 *** [ 2.647529] pstore: using deflate compression [ 2.651383] Platform Keyring initialized [ 2.766642] NET: Registered protocol family 38 [ 2.769332] Key type asymmetric registered [ 2.771771] Asymmetric key parser 'x509' registered [ 2.775055] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.779183] io scheduler mq-deadline registered [ 2.781266] io scheduler kyber registered [ 2.782624] io scheduler bfq registered [ 2.784102] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.788093] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.790736] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.793775] ACPI: Power Button [PWRF] [ 2.799522] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.806220] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.824316] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.853015] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.882598] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.886617] Non-volatile memory driver v1.3 [ 2.887919] Linux agpgart interface v0.103 [ 2.929454] virtio_blk virtio1: [vda] 134032 512-byte logical blocks (68.6 MB/65.4 MiB) [ 2.933791] vda: detected capacity change from 0 to 68624384 [ 2.948956] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 2.951854] vdb: detected capacity change from 0 to 1073741824 [ 2.957937] libphy: Fixed MDIO Bus: probed [ 2.971631] usbcore: registered new interface driver usbserial_generic [ 2.974018] usbserial: USB Serial support registered for generic [ 2.975903] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 2.980146] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 2.981981] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 2.984710] mousedev: PS/2 mouse device common for all mice [ 2.987421] rtc_cmos 00:05: RTC can wake from S4 [ 2.989737] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 2.990116] rtc_cmos 00:05: registered as rtc0 [ 2.993544] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 2.994184] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 2.997560] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 2.999853] intel_pstate: CPU model not supported [ 3.006412] hid: raw HID events driver (C) Jiri Kosina [ 3.009147] usbcore: registered new interface driver usbhid [ 3.011267] usbhid: USB HID core driver [ 3.013084] drop_monitor: Initializing network drop monitor service [ 3.015772] Initializing XFRM netlink socket [ 3.018126] NET: Registered protocol family 10 [ 3.021493] Segment Routing with IPv6 [ 3.023091] NET: Registered protocol family 17 [ 3.026089] mpls_gso: MPLS GSO support [ 3.040132] RAS: Correctable Errors collector initialized. [ 3.042701] AVX version of gcm_enc/dec engaged. [ 3.045375] AES CTR mode by8 optimization enabled [ 3.136886] sched_clock: Marking stable (3136851153, 0)->(3901807569, -764956416) [ 3.141614] registered taskstats version 1 [ 3.144154] Loading compiled-in X.509 certificates [ 3.146613] zswap: loaded using pool lzo/zbud [ 3.171811] Key type big_key registered [ 3.186881] Key type encrypted registered [ 3.189619] ima: No TPM chip found, activating TPM-bypass! [ 3.191414] ima: Allocated hash algorithm: sha1 [ 3.193113] ima: No architecture policies found [ 3.194956] evm: Initialising EVM extended attributes: [ 3.197076] evm: security.selinux [ 3.198451] evm: security.ima [ 3.199675] evm: security.capability [ 3.201292] evm: HMAC attrs: 0x1 [ 3.203934] rtc_cmos 00:05: setting system clock to 2026-01-22 09:37:06 UTC (1769074626) [ 3.208565] debug: unmapping init [mem 0xffffffffabc03000-0xffffffffabdfffff] [ 3.211784] debug: unmapping init [mem 0xffffffffaa982000-0xffffffffaac58fff] [ 3.221118] Write protecting the kernel read-only data: 28672k [ 3.224580] debug: unmapping init [mem 0xffffffffa9003000-0xffffffffa91fffff] [ 3.226526] debug: unmapping init [mem 0xffffffffa9914000-0xffffffffa99fffff] [ 3.263668] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.271783] systemd[1]: Detected virtualization kvm. [ 3.273730] systemd[1]: Detected architecture x86-64. [ 3.275610] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.303184] systemd[1]: No hostname configured. [ 3.304893] systemd[1]: Set hostname to . [ 3.307075] random: systemd: uninitialized urandom read (16 bytes read) [ 3.309540] systemd[1]: Initializing machine ID from random generator. [ 3.342076] random: ln: uninitialized urandom read (6 bytes read) [ 3.452424] random: systemd: uninitialized urandom read (16 bytes read) [ 3.455391] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 3.462941] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 3.469716] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Paths. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Swap. [ OK ] Listening on udev Control Socket. [ OK ] Listening on Journal Socket. Starting Setup Virtual Console... Starting Create Volatile Files and Directories... [ OK ] Reached target Sockets. Starting Create list of required st…ce nodes for the current kernel... Starting Apply Kernel Variables... Starting Journal Service... [ OK ] Reached target Slices. [ OK ] Reached target Timers. [ OK ] Started Memstrack Anylazing Service. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.121295] device-mapper: uevent: version 1.0.3 [ 4.123569] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... [ 4.742697] random: fast init done Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 4.904533] virtio_net virtio0 ens2: renamed from eth0 [ 4.945942] scsi host0: ata_piix [ 4.974902] scsi host1: ata_piix [ 4.977045] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 4.979721] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 9.350043] dracut-initqueue[593]: RTNETLINK answers: File exists [ 9.707301] random: crng init done [ 9.708668] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.206347] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Swap. [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.540817] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.808756] SELinux: Disabled at runtime. [ 11.873351] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.881282] systemd[1]: Detected virtualization kvm. [ 11.883089] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.404800] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.408822] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.414444] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.418920] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.423096] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.431582] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.448894] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. Mounting Kernel Debug File System... Mounting Huge Pages File System... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Reached target rpc_pipefs.target. Starting Apply Kernel Variables... Mounting POSIX Message Queue File System... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on udev Control Socket. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. Starting Remount Root and Kernel File Systems... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice system-getty.slice. [ OK ] Created slice system-serial\x2dgetty.slice. [ 12.616384] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on Process Core Dump Socket. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice system-sshd\x2dkeygen.slice. Starting udev Coldplug all Devices... [ OK ] Started Journal Service. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted POSIX Message Queue File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 12.924419] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Coldplug all Devices. [ OK ] Started udev Kernel Device Manager. [ 13.233988] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.286371] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.388821] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.400610] EDAC sbridge: Ver: 1.1.2 [ 14.556138] Key type dns_resolver registered [ 14.879111] NFS: Registering the id_resolver key type [ 14.881233] Key type id_resolver registered [ 14.883073] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. [ OK ] Started D-Bus System Message Bus. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target sshd-keygen.target. Starting Login Service... Starting Restore /run/initramfs on shutdown... [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg150-client login: [ 28.939170] hrtimer: interrupt took 2396847 ns [ 75.467100] libcfs: loading out-of-tree module taints kernel. [ 75.787463] Key type ._llcrypt registered [ 75.791552] Key type .llcrypt registered [ 76.196281] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 76.225949] alg: No test for adler32 (adler32-zlib) [ 77.889486] Lustre: Lustre: Build Version: 2.17.50_3_gca5b8be [ 78.643784] LNet: Added LNI 192.168.201.50@tcp [8/256/0/180] [ 80.399154] Key type lgssc registered [ 82.080193] Lustre: Echo OBD driver; http://www.lustre.org/ [ 219.589230] Lustre: Mounted lustre-client [ 225.717547] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 245.225212] Lustre: lustre-OST0000-osc-ffff9cc290df1800: disconnect after 23s idle [ 247.067611] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing check_logdir /tmp/testlogs/ [ 251.403565] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing yml_node [ 256.401085] Lustre: DEBUG MARKER: Client: 2.17.50.3 [ 259.548814] Lustre: DEBUG MARKER: MDS: 2.17.50.3 [ 263.211670] Lustre: DEBUG MARKER: OSS: 2.17.50.3 [ 265.037364] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Thu Jan 22 04:41:26 EST 2026 [ 282.494512] LustreError: 7938:0:(lov_obd.c:783:lov_cleanup()) lustre-clilov-ffff9cc290df1800: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 282.500744] LustreError: 7938:0:(obd_class.h:479:obd_check_dev()) Device 5 not setup [ 282.534280] Lustre: Unmounted lustre-client [ 304.637466] Key type lgssc unregistered [ 305.113972] LNet: 8386:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 305.122280] LNetError: 8386:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 305.135917] LNet: Removed LNI 192.168.201.50@tcp [ 305.829148] Key type .llcrypt unregistered [ 305.834424] Key type ._llcrypt unregistered [ 332.821336] Key type ._llcrypt registered [ 332.822609] Key type .llcrypt registered [ 333.146314] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 333.184621] alg: No test for adler32 (adler32-zlib) [ 334.112876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 334.156511] LNet: Added LNI 192.168.201.50@tcp [8/256/0/180] [ 334.173383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 337.818932] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 338.018955] LNet: 9261:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 338.032814] LNetError: 9261:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 338.049978] LNet: Removed LNI 192.168.201.50@tcp [ 338.630578] Key type .llcrypt unregistered [ 338.634118] Key type ._llcrypt unregistered [ 344.502910] Key type ._llcrypt registered [ 344.505338] Key type .llcrypt registered [ 346.101672] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 04:42:47 (1769074967) [ 346.602406] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 346.630398] alg: No test for adler32 (adler32-zlib) [ 347.641162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 347.791182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 347.833644] LNet: 9668:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 347.883650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 347.965492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 347.990907] LNet: 9675:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 355.397724] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 04:42:57 (1769074977) [ 355.654427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 355.684928] LNet: 10252:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 355.711204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 355.799641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 362.057112] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 04:43:03 (1769074983) [ 362.188461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 362.212727] LNet: 10832:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 362.245440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 362.340669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 368.984035] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 04:43:10 (1769074990) [ 369.336351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 369.355524] LNet: 11423:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 369.378954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 369.473859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 376.885896] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 04:43:18 (1769074998) [ 377.043049] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 377.062282] LNet: 12006:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 377.084633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 377.185388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 377.782325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 384.463641] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 04:43:26 (1769075006) [ 384.619931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 384.694177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 384.766265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 385.196134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 385.835520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 392.443972] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 04:43:34 (1769075014) [ 392.726702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 392.750240] LNet: 13197:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 392.758187] LNet: 13197:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 392.786092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 392.877616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 398.842248] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 04:43:40 (1769075020) [ 399.033135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 399.152849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 399.234653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 399.261055] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 399.550417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 399.574658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 399.938895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 399.976543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 400.363957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 400.413996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 400.824305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 400.924194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 401.615669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 401.647042] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 402.113952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 402.152625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 402.497640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 402.536540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 402.987991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 403.029545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 403.552476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 403.599906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 404.195111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 404.238254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 404.510860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 404.542913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 404.883342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 404.922929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 412.244749] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 04:43:54 (1769075034) [ 412.380944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 412.432410] LNet: 14458:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 412.448759] LNet: 14458:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 412.482311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 412.616844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 412.646800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 419.995928] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 04:44:01 (1769075041) [ 420.112070] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 420.161666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 420.263540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 420.291315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 427.339263] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 04:44:09 (1769075049) [ 427.542589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 427.579203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 427.642761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 427.687691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 434.970727] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 04:44:16 (1769075056) [ 435.177290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 435.211353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 435.293861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 435.343129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 443.308572] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 04:44:24 (1769075064) [ 443.681159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 443.744225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 443.885885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 443.970530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 452.912362] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 04:44:34 (1769075074) [ 453.341299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 453.397414] LNet: 17396:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 453.406697] LNet: 17396:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 453.429691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 453.500039] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 453.536456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 460.471415] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 04:44:42 (1769075082) [ 460.693613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 460.772907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 460.853386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 461.395042] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 468.185868] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 04:44:50 (1769075090) [ 468.416580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 468.495809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 468.854854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 469.173669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 476.426994] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 04:44:58 (1769075098) [ 476.627844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 476.711488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 476.857699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 477.392325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 484.479701] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 04:45:06 (1769075106) [ 484.735619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 484.836993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 485.752571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 486.139179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 494.823571] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 04:45:16 (1769075116) [ 495.178904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 495.235947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 495.383685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 495.469691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 503.318702] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 04:45:25 (1769075125) [ 503.416895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 503.493686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 503.606828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 503.695340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 511.341688] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 04:45:33 (1769075133) [ 511.519679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 511.612849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 511.711353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 520.192180] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 04:45:41 (1769075141) [ 520.364444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 520.426446] LNet: 22246:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 520.448052] LNet: 22246:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 7 previous similar messages [ 520.473931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 520.623248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 529.212812] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 04:45:50 (1769075150) [ 529.356777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 529.408851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 529.600768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 530.276040] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 538.112581] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 04:45:59 (1769075159) [ 538.303630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 538.334610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 538.476943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 538.530508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 545.882510] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 04:46:07 (1769075167) [ 546.357191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 546.409802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 546.522759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 546.570463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 554.827800] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 04:46:16 (1769075176) [ 554.933600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 554.989522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 555.138798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 555.190982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 564.111714] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 04:46:25 (1769075185) [ 564.253139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 564.346853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 564.496647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 564.582519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 571.983441] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 04:46:33 (1769075193) [ 572.358157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 572.412231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 572.487120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 572.586577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 572.614801] LNetError: 25785:0:(peer.c:2225:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 572.662764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 572.712335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 579.930889] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 04:46:41 (1769075201) [ 580.054845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 580.105312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 580.285420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.YPxkyJ1SZl [ 580.382746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.YPxkyJ1SZl [ 580.452835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.YPxkyJ1SZl [ 588.293470] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 04:46:49 (1769075209) [ 588.691373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 588.741972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 588.846928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 588.906980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 588.988140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 597.135866] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 04:46:58 (1769075218) [ 597.714367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 597.773420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 597.925433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 602.158490] LNet: Message forwarding will be enabled in 2s [ 605.215144] LNet: Message forwarding enabled [ 605.828491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 605.862452] LNet: Message forwarding will stop in 2s to 6s [ 610.463689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 611.381498] LNet: Message forwarding disabled [ 613.067268] LNet: Message forwarding will be enabled in 2s [ 616.545727] LNet: Message forwarding enabled [ 621.638655] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 04:47:23 (1769075243) [ 621.937140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 623.789650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 623.880539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 629.462864] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 04:47:31 (1769075251) [ 629.776298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 629.814695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 629.890419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 629.927419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 629.972672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 630.023512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 630.067807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 630.094974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 630.121025] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 630.157432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 630.194451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 630.241523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 630.285434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 630.329784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 630.361443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 630.419574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 630.462737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 630.493561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 630.525280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 630.561880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 630.607799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 630.626558] LNetError: 28859:0:(router.c:629:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 637.015517] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 04:47:38 (1769075258) [ 637.164291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 637.262440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 637.377873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 646.177973] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 04:47:47 (1769075267) [ 648.468878] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg150-server [ 650.428711] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 04:47:52 (1769075272) [ 650.603187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 650.665797] LNet: 30357:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 650.675553] LNet: 30357:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 12 previous similar messages [ 650.704176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 650.843099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 650.889363] LNet: Added LNI 192.168.201.50@tcp [8/256/0/180] [ 652.670147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.201.[51-55]@tcp [ 652.688622] LNet: 30577:0:(router.c:733:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 652.746115] LNetError: 30365:0:(router.c:398:lnet_router_discovery_ping_reply()) Peer 192.168.201.52@tcp is being used as a gateway but routing feature is not turned on [ 652.758813] LNetError: 30365:0:(lib-lnet.h:1337:lnet_set_route_aliveness()) route to tcp1 through 192.168.201.52@tcp has gone from up to down [ 653.735130] LNetError: 30365:0:(router.c:398:lnet_router_discovery_ping_reply()) Peer 192.168.201.55@tcp is being used as a gateway but routing feature is not turned on [ 653.743264] LNetError: 30365:0:(router.c:398:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 654.759146] LNetError: 30365:0:(router.c:398:lnet_router_discovery_ping_reply()) Peer 192.168.201.52@tcp is being used as a gateway but routing feature is not turned on [ 654.777444] LNetError: 30365:0:(router.c:398:lnet_router_discovery_ping_reply()) Skipped 4 previous similar messages [ 656.822125] LNetError: 30365:0:(router.c:398:lnet_router_discovery_ping_reply()) Peer 192.168.201.52@tcp is being used as a gateway but routing feature is not turned on [ 656.858253] LNetError: 30365:0:(router.c:398:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 660.400304] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 04:48:02 (1769075282) [ 660.904261] LNetError: 30365:0:(router.c:398:lnet_router_discovery_ping_reply()) Peer 192.168.201.54@tcp is being used as a gateway but routing feature is not turned on [ 660.919818] LNetError: 30365:0:(router.c:398:lnet_router_discovery_ping_reply()) Skipped 19 previous similar messages [ 662.278781] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg150-server [ 663.955522] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 04:48:05 (1769075285) [ 664.022915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 664.067165] LNetError: 31505:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 664.096373] LNet: Removed LNI 192.168.201.50@tcp [ 664.128591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 664.207940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 664.241932] LNet: Added LNI 192.168.201.50@tcp [8/256/0/180] [ 665.036118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.201.[51-56/2]@tcp [ 665.068269] LNet: 31630:0:(router.c:733:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 665.077953] LNet: 31630:0:(router.c:733:lnet_add_route()) Skipped 4 previous similar messages [ 665.107072] LNetError: 31516:0:(lib-lnet.h:1337:lnet_set_route_aliveness()) route to tcp103 through 192.168.201.51@tcp has gone from up to down [ 665.123388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.201.[51-56/2]@tcp [ 665.129140] LNetError: 31516:0:(lib-lnet.h:1337:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 672.302785] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 04:48:14 (1769075294) [ 672.733138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 672.775923] LNetError: 32223:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 672.811816] LNet: Removed LNI 192.168.201.50@tcp [ 672.840486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 672.989535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 673.056638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 673.139814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 673.236478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 673.322731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 673.395145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 673.439755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 673.578925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 673.624664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 673.761227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 673.883374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 673.955904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 674.017670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 674.097052] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 674.138677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 674.217908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 674.318358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 674.354612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 674.409969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 674.507742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 674.550160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 674.620408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 674.702868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 674.766692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 674.835201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 674.958575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 675.014733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 675.111232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 675.236852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 675.273587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 675.328479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 675.408502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 675.468430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 675.589290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 683.292110] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 04:48:25 (1769075305) [ 685.646558] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg150-server [ 687.866752] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 04:48:29 (1769075309) [ 689.745826] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg150-server [ 691.517986] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 04:48:33 (1769075313) [ 692.442787] Key type .llcrypt unregistered [ 692.445384] Key type ._llcrypt unregistered [ 699.496290] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 699.595550] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 699.601218] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 699.610036] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 702.403857] Key type ._llcrypt registered [ 702.406016] Key type .llcrypt registered [ 702.649172] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 702.677364] alg: No test for adler32 (adler32-zlib) [ 703.691208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 703.742566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 703.778859] LNet: Added LNI 192.168.201.50@tcp [8/256/0/180] [ 704.578953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 704.595610] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 706.385046] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 706.411229] LNet: Removed LNI 10.1.2.3@tcp [ 706.534178] LNet: 34343:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 706.539702] LNetError: 34343:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 706.559047] LNet: Removed LNI 192.168.201.50@tcp [ 707.153617] Key type .llcrypt unregistered [ 707.156369] Key type ._llcrypt unregistered [ 719.616384] Key type ._llcrypt registered [ 719.619333] Key type .llcrypt registered [ 720.994679] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 04:49:02 (1769075342) [ 721.596352] Key type .llcrypt unregistered [ 721.601157] Key type ._llcrypt unregistered [ 727.661611] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 727.882405] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 727.889331] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 727.896933] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 730.551402] Key type ._llcrypt registered [ 730.555309] Key type .llcrypt registered [ 730.846642] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 730.880130] alg: No test for adler32 (adler32-zlib) [ 731.798320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 731.864334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 731.883403] LNet: Added LNI 192.168.201.50@tcp [8/256/0/180] [ 732.482356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 732.498776] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 734.948666] LNet: 35831:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 734.955915] LNetError: 35831:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 734.960770] LNet: Removed LNI 192.168.201.50@tcp [ 734.975617] LNet: Removed LNI 10.1.2.3@tcp [ 735.549649] Key type .llcrypt unregistered [ 735.552521] Key type ._llcrypt unregistered [ 751.177847] Key type ._llcrypt registered [ 751.182633] Key type .llcrypt registered [ 752.532499] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 04:49:34 (1769075374) [ 753.315492] Key type .llcrypt unregistered [ 753.322627] Key type ._llcrypt unregistered [ 760.892328] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 761.122389] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 761.127557] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 761.133557] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 764.024460] Key type ._llcrypt registered [ 764.026841] Key type .llcrypt registered [ 764.303317] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 764.336131] alg: No test for adler32 (adler32-zlib) [ 765.376880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 765.463593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 765.499897] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 768.121497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 768.169434] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 771.020657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 771.049629] LNet: Removed LNI 10.1.2.3@tcp [ 771.075371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 771.105992] LNet: Removed LNI 10.1.2.31@tcp [ 771.122502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 771.159286] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 772.010418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 772.038976] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 773.372345] LNet: 37747:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 773.379144] LNetError: 37747:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 773.385296] LNet: Removed LNI 10.1.2.3@tcp [ 773.395184] LNet: Removed LNI 10.1.2.31@tcp [ 774.037542] Key type .llcrypt unregistered [ 774.043937] Key type ._llcrypt unregistered [ 787.186325] Key type ._llcrypt registered [ 787.187725] Key type .llcrypt registered [ 788.486456] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 04:50:10 (1769075410) [ 789.000945] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 789.024371] alg: No test for adler32 (adler32-zlib) [ 790.099702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 790.174276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 790.218683] LNet: Added LNI 192.168.201.50@tcp [8/256/0/180] [ 791.385600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 791.408648] LNet: 38697:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 791.413499] LNetError: 38697:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 791.446866] LNet: Removed LNI 192.168.201.50@tcp [ 791.466277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 791.512698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.201.50@tcp [ 791.541103] LNet: Added LNI 192.168.201.50@tcp [8/256/0/180] [ 792.360765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.201.50@tcp [ 792.394480] LNet: Removed LNI 192.168.201.50@tcp [ 792.469600] LNet: 38832:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 793.178623] Key type .llcrypt unregistered [ 793.183432] Key type ._llcrypt unregistered [ 806.382374] Key type ._llcrypt registered [ 806.384641] Key type .llcrypt registered [ 808.434153] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 04:50:29 (1769075429) [ 810.409642] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg150-server [ 812.405445] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 04:50:34 (1769075434) [ 813.359309] Key type .llcrypt unregistered [ 813.362249] Key type ._llcrypt unregistered [ 820.744960] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 820.924029] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 820.933815] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 820.943172] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 823.868938] Key type ._llcrypt registered [ 823.870579] Key type .llcrypt registered [ 824.090796] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 824.103666] alg: No test for adler32 (adler32-zlib) [ 825.069212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 825.143841] LNet: Added LNI 192.168.201.50@tcp1 [8/256/0/180] [ 825.968987] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 828.981520] LNet: 40666:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 828.991319] LNetError: 40666:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 828.998725] LNet: Removed LNI 192.168.201.50@tcp1 [ 829.019322] LNet: Removed LNI 10.1.2.3@tcp2 [ 830.149441] Key type .llcrypt unregistered [ 830.154092] Key type ._llcrypt unregistered [ 845.557218] Key type ._llcrypt registered [ 845.561587] Key type .llcrypt registered [ 847.623361] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 04:51:08 (1769075468) [ 848.104162] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 848.124104] alg: No test for adler32 (adler32-zlib) [ 849.065026] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 849.125818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 849.151332] LNet: Added LNI 192.168.201.50@tcp [8/256/0/180] [ 850.361334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 850.402393] LNet: 41634:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 850.410318] LNetError: 41634:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 850.425399] LNet: Removed LNI 192.168.201.50@tcp [ 850.468941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 850.613500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 850.686933] LNet: Added LNI 192.168.201.50@tcp [8/256/0/180] [ 859.417915] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 04:51:20 (1769075480) [ 859.702772] LNet: 42226:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 859.732658] LNetError: 42226:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 859.810618] LNet: Removed LNI 192.168.201.50@tcp [ 860.583308] Key type .llcrypt unregistered [ 860.589384] Key type ._llcrypt unregistered [ 866.568434] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 866.726767] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 866.740571] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 866.756871] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 869.628237] Key type ._llcrypt registered [ 869.631963] Key type .llcrypt registered [ 869.986768] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 870.015662] alg: No test for adler32 (adler32-zlib) [ 871.002239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 871.083459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 871.137292] LNet: Added LNI 192.168.201.50@tcp [8/256/0/180] [ 871.961370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 871.977074] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 874.840651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 874.867565] LNet: 42967:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 874.875847] LNetError: 42967:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 874.880473] LNet: Removed LNI 192.168.201.50@tcp [ 874.894158] LNet: Removed LNI 10.1.2.3@tcp [ 874.914925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 874.990544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 875.049767] LNet: Added LNI 192.168.201.50@tcp [8/256/0/180] [ 875.057043] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 875.901791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 875.946446] LNet: 42994:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 875.966150] LNetError: 42994:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 875.979055] LNetError: 42994:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 876.000050] LNet: Removed LNI 192.168.201.50@tcp [ 876.029156] LNet: Removed LNI 10.1.2.3@tcp [ 876.080562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 876.246246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 876.277435] LNet: Added LNI 192.168.201.50@tcp [8/256/0/180] [ 876.288854] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 876.909993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 876.959438] LNet: Added LNI 192.168.201.50@tcp2 [8/256/0/180] [ 878.160300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 878.177149] LNet: 43157:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 878.185390] LNetError: 43157:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 878.190253] LNetError: 43157:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 878.194653] LNet: Removed LNI 192.168.201.50@tcp [ 878.195814] LNet: Removed LNI 10.1.2.3@tcp [ 878.210677] LNet: Removed LNI 192.168.201.50@tcp2 [ 878.230887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 878.288779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 878.311568] LNet: Added LNI 192.168.201.50@tcp [8/256/0/180] [ 878.317089] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 878.322751] LNet: Added LNI 192.168.201.50@tcp2 [8/256/0/180] [ 886.476988] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 04:51:47 (1769075507) [ 886.613887] LNet: 43748:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 886.630978] LNetError: 43748:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 886.642058] LNetError: 43748:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 886.655656] LNet: Removed LNI 192.168.201.50@tcp [ 886.657402] LNet: Removed LNI 10.1.2.3@tcp [ 886.706407] LNet: Removed LNI 192.168.201.50@tcp2 [ 887.531204] Key type .llcrypt unregistered [ 887.532856] Key type ._llcrypt unregistered [ 894.320993] Key type ._llcrypt registered [ 894.322861] Key type .llcrypt registered [ 894.604984] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 894.642282] alg: No test for adler32 (adler32-zlib) [ 895.768710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 895.878127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 895.912315] LNet: Added LNI 192.168.201.50@tcp [8/256/0/180] [ 897.063935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 897.094074] LNet: 44145:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 897.099049] LNetError: 44145:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 897.139115] LNet: Removed LNI 192.168.201.50@tcp [ 897.181142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 897.329618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 897.375617] LNet: Added LNI 192.168.201.50@tcp [8/256/0/180] [ 898.051464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 898.087167] LNet: 44176:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 898.107308] LNetError: 44176:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 898.125854] LNet: Removed LNI 192.168.201.50@tcp [ 898.154946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 898.242537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 898.281663] LNet: Added LNI 192.168.201.50@tcp [8/256/0/180] [ 899.139380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 899.167847] LNet: 44206:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 899.178858] LNetError: 44206:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 899.197840] LNet: Removed LNI 192.168.201.50@tcp [ 899.215288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 899.314935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 899.352756] LNet: Added LNI 192.168.201.50@tcp [8/256/0/180] [ 899.939617] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 900.064879] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 900.081474] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 900.085368] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 901.648898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 901.674608] LNet: 44278:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 901.693939] LNetError: 44278:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 901.716955] LNet: Removed LNI 192.168.201.50@tcp [ 901.777219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 901.920912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 901.961970] LNet: Added LNI 192.168.201.50@tcp [8/256/0/180] [ 910.761875] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 04:52:12 (1769075532) [ 910.919448] LNet: 44868:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 910.928682] LNetError: 44868:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 910.952864] LNet: Removed LNI 192.168.201.50@tcp [ 911.662315] Key type .llcrypt unregistered [ 911.664248] Key type ._llcrypt unregistered [ 918.389834] Key type ._llcrypt registered [ 918.397606] Key type .llcrypt registered [ 918.781967] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 918.810871] alg: No test for adler32 (adler32-zlib) [ 919.781102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 919.824769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 919.841018] LNet: Added LNI 192.168.201.50@tcp [8/256/0/180] [ 920.654458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 920.676971] LNet: 45271:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 920.683199] LNetError: 45271:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 920.698780] LNet: Removed LNI 192.168.201.50@tcp [ 920.774253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 920.864309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 920.907178] LNet: Added LNI 192.168.201.50@tcp [8/256/0/180] [ 929.078773] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 04:52:30 (1769075550) [ 929.214087] LNet: 45856:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 929.229754] LNetError: 45856:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 929.252750] LNet: Removed LNI 192.168.201.50@tcp [ 930.042823] Key type .llcrypt unregistered [ 930.047990] Key type ._llcrypt unregistered [ 938.226214] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 938.469739] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 938.483364] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 938.486157] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 939.885321] Key type ._llcrypt registered [ 939.887158] Key type .llcrypt registered [ 940.031936] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 940.051528] alg: No test for adler32 (adler32-zlib) [ 941.036242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 941.102782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 941.147323] LNet: Added LNI 192.168.201.50@tcp [8/256/0/180] [ 942.142218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 942.164721] LNet: 46302:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 942.169194] LNetError: 46302:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 942.183112] LNet: Removed LNI 192.168.201.50@tcp [ 942.201216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 942.264304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 942.285036] LNet: Added LNI 192.168.201.50@tcp [8/256/0/180] [ 949.914202] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 04:52:51 (1769075571) [ 950.089312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 950.126591] LNet: 46898:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 950.142026] LNetError: 46898:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 950.167630] LNet: Removed LNI 192.168.201.50@tcp [ 950.219295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 950.335553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 956.884532] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 04:52:58 (1769075578) [ 957.020834] LNet: 47475:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 957.768642] Key type .llcrypt unregistered [ 957.771937] Key type ._llcrypt unregistered [ 965.570527] Key type ._llcrypt registered [ 965.586135] Key type .llcrypt registered [ 966.049220] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 966.088579] alg: No test for adler32 (adler32-zlib) [ 967.141095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 967.248289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 967.310581] LNet: Added LNI 192.168.201.50@tcp [8/256/0/180] [ 968.921802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 968.974429] LNet: 47877:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 968.987165] LNetError: 47877:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 969.008977] LNet: Removed LNI 192.168.201.50@tcp [ 969.031209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 969.136374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 969.180838] LNet: Added LNI 192.168.201.50@tcp [8/256/0/180] [ 976.723556] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 04:53:18 (1769075598) [ 977.053934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 977.084102] LNet: 48472:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 977.098000] LNetError: 48472:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 977.131326] LNet: Removed LNI 192.168.201.50@tcp [ 977.179334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 977.285851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 984.440977] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 04:53:25 (1769075605) [ 986.487729] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 988.172454] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 04:53:30 (1769075610) [ 989.589487] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 992.023363] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 04:53:33 (1769075613) [ 993.559951] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 995.600982] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 04:53:37 (1769075617) [ 997.402557] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 999.690886] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 04:53:41 (1769075621) [ 1001.427068] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 1003.421055] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 04:53:45 (1769075625) [ 1003.667288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1003.683426] LNet: 50526:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1003.706823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1003.750977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1003.782274] LNet: Added LNI 192.168.201.50@tcp [8/256/0/180] [ 1004.419099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.201.50 [ 1004.431356] LNet: 50659:0:(config.c:1575:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1004.437936] LNet: Removed LNI 192.168.201.50@tcp [ 1011.161423] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 04:53:53 (1769075633) [ 1011.316240] LNet: 51222:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1011.342537] LNetError: 51222:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1011.388309] LNet: Removed LNI 192.168.201.50@tcp [ 1011.916897] Key type .llcrypt unregistered [ 1011.920217] Key type ._llcrypt unregistered [ 1018.961667] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1019.358383] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1019.364153] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1019.370271] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1022.343401] Key type ._llcrypt registered [ 1022.347886] Key type .llcrypt registered [ 1022.589225] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1022.608431] alg: No test for adler32 (adler32-zlib) [ 1024.192579] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1030.562779] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 04:54:12 (1769075652) [ 1030.948623] LNet: 52132:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1030.953472] LNetError: 52132:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1030.967241] LNet: Removed LNI 10.1.2.3@tcp [ 1031.788867] Key type .llcrypt unregistered [ 1031.790709] Key type ._llcrypt unregistered [ 1038.795200] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1039.198124] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1039.215649] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1039.224785] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1041.079794] Key type ._llcrypt registered [ 1041.084492] Key type .llcrypt registered [ 1041.287568] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1041.316774] alg: No test for adler32 (adler32-zlib) [ 1042.418739] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1049.266195] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 04:54:30 (1769075670) [ 1049.468042] LNet: 53008:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1049.484242] LNetError: 53008:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1049.549192] LNet: Removed LNI 10.1.2.3@tcp [ 1050.383679] Key type .llcrypt unregistered [ 1050.386953] Key type ._llcrypt unregistered [ 1056.752255] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1057.038731] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1057.050703] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1057.062931] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1059.938403] Key type ._llcrypt registered [ 1059.940520] Key type .llcrypt registered [ 1060.235605] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1060.274261] alg: No test for adler32 (adler32-zlib) [ 1061.271196] LNet: 53310:0:(config.c:1575:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1061.300308] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1068.331991] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 04:54:50 (1769075690) [ 1068.443089] LNet: 53897:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1068.448084] LNetError: 53897:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1068.472172] LNet: Removed LNI 10.1.2.3@tcp [ 1068.860515] Key type .llcrypt unregistered [ 1068.869328] Key type ._llcrypt unregistered [ 1075.627948] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1075.919359] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1075.927663] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1075.934301] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1078.776384] Key type ._llcrypt registered [ 1078.779392] Key type .llcrypt registered [ 1078.920364] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1078.936348] alg: No test for adler32 (adler32-zlib) [ 1079.945175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1080.014257] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1088.505391] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 04:55:10 (1769075710) [ 1088.966506] LNet: 55129:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1088.972516] LNetError: 55129:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1089.006596] LNet: Removed LNI 10.1.2.3@tcp [ 1089.565979] Key type .llcrypt unregistered [ 1089.568300] Key type ._llcrypt unregistered [ 1095.385316] Key type ._llcrypt registered [ 1095.387522] Key type .llcrypt registered [ 1095.722722] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1095.763654] alg: No test for adler32 (adler32-zlib) [ 1096.715932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1096.751901] LNet: Added LNI 192.168.201.50@tcp [8/256/0/180] [ 1104.983072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.150@tcp [ 1105.915097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.50@tcp -d 192.168.201.150@tcp -m GET -e local_interrupt -r 1 [ 1105.964760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.50@tcp -d 192.168.201.50@tcp -m GET -e local_interrupt -r 1 [ 1106.013073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.150@tcp [ 1106.032142] LNet: There was an unexpected network error while writing to 192.168.201.150: rc = -22 [ 1106.311663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1106.345837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1106.396024] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1106.437238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1106.496480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1106.551943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1106.603214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1106.648662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1106.664905] LNet: 55563:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1106.683972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1106.710981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1106.739676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1106.779479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1106.810370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1106.840154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1106.894393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1107.458493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.50@tcp -d 192.168.201.150@tcp -m GET -e local_dropped -r 1 [ 1107.497905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.50@tcp -d 192.168.201.50@tcp -m GET -e local_dropped -r 1 [ 1107.524987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.150@tcp [ 1107.540101] LNet: There was an unexpected network error while writing to 192.168.201.150: rc = -22 [ 1107.709616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1107.735835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1107.762796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1107.792263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1107.817794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1107.845724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1107.871507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1107.905812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1107.918795] LNet: 55673:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1107.937497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1107.968821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1108.006272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1108.034761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1108.067175] LNet: 1 local NIs in recovery (showing 1): 192.168.201.50@tcp [ 1108.073946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1108.105878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1108.162806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1108.911140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.50@tcp -d 192.168.201.150@tcp -m GET -e local_aborted -r 1 [ 1108.974788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.50@tcp -d 192.168.201.50@tcp -m GET -e local_aborted -r 1 [ 1109.027406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.150@tcp [ 1109.056273] LNet: There was an unexpected network error while writing to 192.168.201.150: rc = -22 [ 1109.060358] LNet: Skipped 1 previous similar message [ 1109.353547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1109.415337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1109.470480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1109.553391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1109.603578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1109.681241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1109.736161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1109.777515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1109.806307] LNet: 55782:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1109.823587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1109.849516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1109.869609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1109.912139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1109.973461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1110.008937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1110.072546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1110.670349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.50@tcp -d 192.168.201.150@tcp -m GET -e local_no_route -r 1 [ 1110.770840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.50@tcp -d 192.168.201.50@tcp -m GET -e local_no_route -r 1 [ 1110.825045] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.150@tcp [ 1111.259246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1111.315800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1111.397623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1111.455916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1111.515811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1111.584799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1111.615878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1111.650529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1111.695840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1111.771875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1111.804344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1111.847026] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1111.881975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1111.917854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1111.968028] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1112.561366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.50@tcp -d 192.168.201.150@tcp -m GET -e local_timeout -r 1 [ 1112.603841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.50@tcp -d 192.168.201.50@tcp -m GET -e local_timeout -r 1 [ 1112.635638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.150@tcp [ 1112.652657] LNet: There was an unexpected network error while writing to 192.168.201.150: rc = -22 [ 1112.659309] LNet: Skipped 1 previous similar message [ 1112.817649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1112.839430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1112.867362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1112.894200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1112.921175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1112.943489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1112.972621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1113.002946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1113.018759] LNet: 56001:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1113.024952] LNet: 56001:0:(api-ni.c:343:recovery_interval_set()) Skipped 1 previous similar message [ 1113.042851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1113.071888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1113.103406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1113.138101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1113.194960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1113.242456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1113.292356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1114.013905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.50@tcp -d 192.168.201.150@tcp -m GET -e local_error -r 1 [ 1114.050960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.50@tcp -d 192.168.201.50@tcp -m GET -e local_error -r 1 [ 1114.073495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.150@tcp [ 1114.299902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1114.371128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1114.408648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1114.437806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1114.515145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1114.565902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1114.597586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1114.623522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1114.651612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1114.682891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1114.723624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1114.767563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1114.846526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1114.886030] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1114.931332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1115.219846] LNet: 56150:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1115.225991] LNetError: 56150:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1115.259221] LNet: Removed LNI 192.168.201.50@tcp [ 1115.889985] Key type .llcrypt unregistered [ 1115.892620] Key type ._llcrypt unregistered [ 1136.950598] Key type ._llcrypt registered [ 1136.954577] Key type .llcrypt registered [ 1138.784036] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 04:55:59 (1769075759) [ 1139.577547] Key type .llcrypt unregistered [ 1139.583315] Key type ._llcrypt unregistered [ 1146.935337] Key type ._llcrypt registered [ 1146.947219] Key type .llcrypt registered [ 1147.232284] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1147.250266] alg: No test for adler32 (adler32-zlib) [ 1148.205483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1148.263061] LNet: Added LNI 192.168.201.50@tcp [8/256/0/180] [ 1157.210714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.150@tcp [ 1168.491714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1168.538907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1168.606225] LNet: Added LNI 192.168.201.50@tcp1 [8/256/0/180] [ 1170.273354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.50@tcp -d 192.168.201.150@tcp -m GET -e local_interrupt -r 1 [ 1170.301972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.50@tcp -d 192.168.201.150@tcp1 -m GET -e local_interrupt -r 1 [ 1170.333586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.50@tcp -d 192.168.201.50@tcp -m GET -e local_interrupt -r 1 [ 1170.372995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.50@tcp -d 192.168.201.50@tcp1 -m GET -e local_interrupt -r 1 [ 1170.437712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.50@tcp1 -d 192.168.201.150@tcp -m GET -e local_interrupt -r 1 [ 1170.487248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.50@tcp1 -d 192.168.201.150@tcp1 -m GET -e local_interrupt -r 1 [ 1170.536943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.50@tcp1 -d 192.168.201.50@tcp -m GET -e local_interrupt -r 1 [ 1170.604882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.50@tcp1 -d 192.168.201.50@tcp1 -m GET -e local_interrupt -r 1 [ 1170.657089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.150@tcp [ 1170.688969] LNet: There was an unexpected network error while writing to 192.168.201.150: rc = -22 [ 1170.708509] LNet: 1 local NIs in recovery (showing 1): 192.168.201.50@tcp [ 1171.045238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1171.091128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1171.137761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1171.182561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1171.232405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1171.284612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1171.329353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1171.371610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1171.401631] LNet: 57614:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1171.429646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1171.469898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1171.541058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1171.604620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1171.665981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1171.705707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1171.739805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1172.334647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.50@tcp -d 192.168.201.150@tcp -m GET -e local_dropped -r 1 [ 1172.372409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.50@tcp -d 192.168.201.150@tcp1 -m GET -e local_dropped -r 1 [ 1172.405485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.50@tcp -d 192.168.201.50@tcp -m GET -e local_dropped -r 1 [ 1172.442602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.50@tcp -d 192.168.201.50@tcp1 -m GET -e local_dropped -r 1 [ 1172.494854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.50@tcp1 -d 192.168.201.150@tcp -m GET -e local_dropped -r 1 [ 1172.539644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.50@tcp1 -d 192.168.201.150@tcp1 -m GET -e local_dropped -r 1 [ 1172.569934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.50@tcp1 -d 192.168.201.50@tcp -m GET -e local_dropped -r 1 [ 1172.603078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.50@tcp1 -d 192.168.201.50@tcp1 -m GET -e local_dropped -r 1 [ 1172.637715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.150@tcp [ 1172.654176] LNet: There was an unexpected network error while writing to 192.168.201.150: rc = -22 [ 1172.659808] LNet: Skipped 2 previous similar messages [ 1172.920570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1172.949885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1172.989852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1173.039712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1173.075853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1173.116782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1173.143385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1173.169602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1173.188579] LNet: 57735:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1173.204651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1173.237878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1173.270380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1173.312023] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1173.347565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1173.378640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1173.406327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1174.038657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.50@tcp -d 192.168.201.150@tcp -m GET -e local_aborted -r 1 [ 1174.083450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.50@tcp -d 192.168.201.150@tcp1 -m GET -e local_aborted -r 1 [ 1174.154864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.50@tcp -d 192.168.201.50@tcp -m GET -e local_aborted -r 1 [ 1174.214847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.50@tcp -d 192.168.201.50@tcp1 -m GET -e local_aborted -r 1 [ 1174.262835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.50@tcp1 -d 192.168.201.150@tcp -m GET -e local_aborted -r 1 [ 1174.297411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.50@tcp1 -d 192.168.201.150@tcp1 -m GET -e local_aborted -r 1 [ 1174.337454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.50@tcp1 -d 192.168.201.50@tcp -m GET -e local_aborted -r 1 [ 1174.366788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.50@tcp1 -d 192.168.201.50@tcp1 -m GET -e local_aborted -r 1 [ 1174.394956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.150@tcp [ 1174.408688] LNet: There was an unexpected network error while writing to 192.168.201.150: rc = -22 [ 1174.412610] LNet: Skipped 2 previous similar messages [ 1174.615950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1174.653389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1174.686368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1174.718597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1174.746993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1174.780605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1174.816584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1174.850324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1174.863074] LNet: 57857:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1174.883610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1174.913944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1174.945433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1174.975500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1175.012541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1175.044885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1175.079579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1175.608871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.50@tcp -d 192.168.201.150@tcp -m GET -e local_no_route -r 1 [ 1175.649452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.50@tcp -d 192.168.201.150@tcp1 -m GET -e local_no_route -r 1 [ 1175.690496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.50@tcp -d 192.168.201.50@tcp -m GET -e local_no_route -r 1 [ 1175.726529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.50@tcp -d 192.168.201.50@tcp1 -m GET -e local_no_route -r 1 [ 1175.759844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.50@tcp1 -d 192.168.201.150@tcp -m GET -e local_no_route -r 1 [ 1175.796144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.50@tcp1 -d 192.168.201.150@tcp1 -m GET -e local_no_route -r 1 [ 1175.839816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.50@tcp1 -d 192.168.201.50@tcp -m GET -e local_no_route -r 1 [ 1175.890647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.50@tcp1 -d 192.168.201.50@tcp1 -m GET -e local_no_route -r 1 [ 1175.949439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.150@tcp [ 1176.313783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1176.366389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1176.408534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1176.438806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1176.470716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1176.500519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1176.536233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1176.561792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1176.591143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1176.620749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1176.646777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1176.676784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1176.708081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1176.731978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1176.767642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1177.251546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.50@tcp -d 192.168.201.150@tcp -m GET -e local_timeout -r 1 [ 1177.287629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.50@tcp -d 192.168.201.150@tcp1 -m GET -e local_timeout -r 1 [ 1177.322847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.50@tcp -d 192.168.201.50@tcp -m GET -e local_timeout -r 1 [ 1177.356743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.50@tcp -d 192.168.201.50@tcp1 -m GET -e local_timeout -r 1 [ 1177.394650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.50@tcp1 -d 192.168.201.150@tcp -m GET -e local_timeout -r 1 [ 1177.439667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.50@tcp1 -d 192.168.201.150@tcp1 -m GET -e local_timeout -r 1 [ 1177.477630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.50@tcp1 -d 192.168.201.50@tcp -m GET -e local_timeout -r 1 [ 1177.516124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.50@tcp1 -d 192.168.201.50@tcp1 -m GET -e local_timeout -r 1 [ 1177.564788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.150@tcp [ 1177.581743] LNet: There was an unexpected network error while writing to 192.168.201.150: rc = -22 [ 1177.588439] LNet: Skipped 5 previous similar messages [ 1177.894572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1177.931451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1177.962522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1177.993348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1178.024639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1178.050117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1178.083768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1178.116531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1178.133472] LNet: 58099:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1178.139355] LNet: 58099:0:(api-ni.c:343:recovery_interval_set()) Skipped 1 previous similar message [ 1178.162766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1178.192797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1178.221522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1178.254878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1178.285379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1178.318367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1178.341625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1178.931458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.50@tcp -d 192.168.201.150@tcp -m GET -e local_error -r 1 [ 1178.972493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.50@tcp -d 192.168.201.150@tcp1 -m GET -e local_error -r 1 [ 1179.039680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.50@tcp -d 192.168.201.50@tcp -m GET -e local_error -r 1 [ 1179.080187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.50@tcp -d 192.168.201.50@tcp1 -m GET -e local_error -r 1 [ 1179.118677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.50@tcp1 -d 192.168.201.150@tcp -m GET -e local_error -r 1 [ 1179.160735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.50@tcp1 -d 192.168.201.150@tcp1 -m GET -e local_error -r 1 [ 1179.201504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.50@tcp1 -d 192.168.201.50@tcp -m GET -e local_error -r 1 [ 1179.244663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.50@tcp1 -d 192.168.201.50@tcp1 -m GET -e local_error -r 1 [ 1179.279337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.150@tcp [ 1179.527561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1179.554758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1179.579732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1179.611608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1179.659215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1179.686624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1179.734333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1179.771231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1179.820479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1179.856730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1179.899507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1179.932731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1179.961399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1180.025400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1180.064430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1185.941975] LNet: 58296:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1185.948380] LNetError: 58296:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1185.955396] LNet: Removed LNI 192.168.201.50@tcp [ 1185.964875] LNet: Removed LNI 192.168.201.50@tcp1 [ 1186.357738] Key type .llcrypt unregistered [ 1186.360402] Key type ._llcrypt unregistered [ 1202.512900] Key type ._llcrypt registered [ 1202.514083] Key type .llcrypt registered [ 1203.670612] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 04:57:05 (1769075825) [ 1204.361775] Key type .llcrypt unregistered [ 1204.368919] Key type ._llcrypt unregistered [ 1211.114586] Key type ._llcrypt registered [ 1211.117301] Key type .llcrypt registered [ 1211.321543] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1211.337435] alg: No test for adler32 (adler32-zlib) [ 1212.314710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1212.339833] LNet: Added LNI 192.168.201.50@tcp [8/256/0/180] [ 1220.262386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.150@tcp [ 1220.912912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.50@tcp -d 192.168.201.150@tcp -m GET -e remote_dropped -r 1 [ 1220.956909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.50@tcp -d 192.168.201.50@tcp -m GET -e remote_dropped -r 1 [ 1220.992732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.150@tcp [ 1221.018124] LNet: There was an unexpected network error while writing to 192.168.201.150: rc = -22 [ 1221.261336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1221.290656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1221.319901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1221.352686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1221.378698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1221.418844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1221.441811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1221.470637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1221.485624] LNet: 59540:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1221.503129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1221.531770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1221.561793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1221.596886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1221.629759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1221.660658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1221.701982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1222.193769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.50@tcp -d 192.168.201.150@tcp -m GET -e remote_error -r 1 [ 1222.233154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.50@tcp -d 192.168.201.50@tcp -m GET -e remote_error -r 1 [ 1222.269226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.150@tcp [ 1222.289496] LNet: There was an unexpected network error while writing to 192.168.201.150: rc = -22 [ 1222.493813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1222.524208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1222.551921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1222.587870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1222.621289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1222.646197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1222.676344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1222.704343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1222.716521] LNet: 59649:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1222.733552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1222.763254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1222.800630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1222.835539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1222.873246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1222.898946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1222.929966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1223.450725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.50@tcp -d 192.168.201.150@tcp -m GET -e remote_timeout -r 1 [ 1223.485397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.50@tcp -d 192.168.201.50@tcp -m GET -e remote_timeout -r 1 [ 1223.525724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.150@tcp [ 1223.552108] LNet: There was an unexpected network error while writing to 192.168.201.150: rc = -22 [ 1223.803415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1223.833049] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1223.870905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1223.919787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1223.952931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1224.002415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1224.035770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1224.075418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1224.102609] LNet: 59759:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1224.141820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1224.181165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1224.232750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1224.284968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1224.326287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1224.379328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1224.423229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1224.741903] LNet: 59799:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1224.749925] LNetError: 59799:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1224.768647] LNet: Removed LNI 192.168.201.50@tcp [ 1225.204948] Key type .llcrypt unregistered [ 1225.208502] Key type ._llcrypt unregistered [ 1243.527769] Key type ._llcrypt registered [ 1243.530273] Key type .llcrypt registered [ 1245.191525] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 04:57:46 (1769075866) [ 1246.139497] Key type .llcrypt unregistered [ 1246.141563] Key type ._llcrypt unregistered [ 1253.059270] Key type ._llcrypt registered [ 1253.064747] Key type .llcrypt registered [ 1253.445884] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1253.488172] alg: No test for adler32 (adler32-zlib) [ 1254.421985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1254.464657] LNet: Added LNI 192.168.201.50@tcp [8/256/0/180] [ 1262.822711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.150@tcp [ 1275.037664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1275.143253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1275.190339] LNet: Added LNI 192.168.201.50@tcp1 [8/256/0/180] [ 1277.217367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.50@tcp -d 192.168.201.150@tcp -m GET -e remote_dropped -r 1 [ 1277.263726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.50@tcp -d 192.168.201.150@tcp1 -m GET -e remote_dropped -r 1 [ 1277.321365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.50@tcp -d 192.168.201.50@tcp -m GET -e remote_dropped -r 1 [ 1277.384914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.50@tcp -d 192.168.201.50@tcp1 -m GET -e remote_dropped -r 1 [ 1277.433878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.50@tcp1 -d 192.168.201.150@tcp -m GET -e remote_dropped -r 1 [ 1277.501967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.50@tcp1 -d 192.168.201.150@tcp1 -m GET -e remote_dropped -r 1 [ 1277.570156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.50@tcp1 -d 192.168.201.50@tcp -m GET -e remote_dropped -r 1 [ 1277.626453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.50@tcp1 -d 192.168.201.50@tcp1 -m GET -e remote_dropped -r 1 [ 1277.720935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.150@tcp [ 1277.765246] LNet: There was an unexpected network error while writing to 192.168.201.150: rc = -22 [ 1277.773160] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.150@tcp [ 1278.033454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1278.096407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1278.187327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1278.228118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1278.273071] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1278.294870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1278.330435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1278.380596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1278.422826] LNet: 61263:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1278.447704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1278.494617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1278.542083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1278.581509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1278.622398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1278.674158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1278.720767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1278.769861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1279.245618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.50@tcp -d 192.168.201.150@tcp -m GET -e remote_error -r 1 [ 1279.271252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.50@tcp -d 192.168.201.150@tcp1 -m GET -e remote_error -r 1 [ 1279.310261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.50@tcp -d 192.168.201.50@tcp -m GET -e remote_error -r 1 [ 1279.368586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.50@tcp -d 192.168.201.50@tcp1 -m GET -e remote_error -r 1 [ 1279.391224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.50@tcp1 -d 192.168.201.150@tcp -m GET -e remote_error -r 1 [ 1279.426305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.50@tcp1 -d 192.168.201.150@tcp1 -m GET -e remote_error -r 1 [ 1279.461062] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.50@tcp1 -d 192.168.201.50@tcp -m GET -e remote_error -r 1 [ 1279.495167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.50@tcp1 -d 192.168.201.50@tcp1 -m GET -e remote_error -r 1 [ 1279.541873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.150@tcp [ 1279.575490] LNet: There was an unexpected network error while writing to 192.168.201.150: rc = -22 [ 1279.580990] LNet: Skipped 2 previous similar messages [ 1279.890639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1279.917511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1279.951891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1279.976128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1280.012391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1280.059781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1280.106535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1280.159393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1280.198585] LNet: 61386:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1280.231207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1280.267166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1280.297440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1280.350446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1280.395822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1280.440972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1280.487781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1280.518986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1281.001350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.50@tcp -d 192.168.201.150@tcp -m GET -e remote_timeout -r 1 [ 1281.043960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.50@tcp -d 192.168.201.150@tcp1 -m GET -e remote_timeout -r 1 [ 1281.077241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.50@tcp -d 192.168.201.50@tcp -m GET -e remote_timeout -r 1 [ 1281.115976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.50@tcp -d 192.168.201.50@tcp1 -m GET -e remote_timeout -r 1 [ 1281.187937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.50@tcp1 -d 192.168.201.150@tcp -m GET -e remote_timeout -r 1 [ 1281.238338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.50@tcp1 -d 192.168.201.150@tcp1 -m GET -e remote_timeout -r 1 [ 1281.330441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.50@tcp1 -d 192.168.201.50@tcp -m GET -e remote_timeout -r 1 [ 1281.355433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.50@tcp1 -d 192.168.201.50@tcp1 -m GET -e remote_timeout -r 1 [ 1281.381875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.150@tcp [ 1281.396776] LNet: There was an unexpected network error while writing to 192.168.201.150: rc = -22 [ 1281.565371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1281.601736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1281.624462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1281.655437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1281.676256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1281.709807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1281.746496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1281.796381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1281.818485] LNet: 61510:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1281.845358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1281.875099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1281.908598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1281.951298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1282.000116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1282.047850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1282.116786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1282.190732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1288.264640] LNet: 61587:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1288.269746] LNetError: 61587:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1289.311178] LNet: Removed LNI 192.168.201.50@tcp [ 1289.313148] LNetError: 61587:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1289.335612] LNet: Removed LNI 192.168.201.50@tcp1 [ 1289.887544] Key type .llcrypt unregistered [ 1289.893653] Key type ._llcrypt unregistered [ 1306.883957] Key type ._llcrypt registered [ 1306.886723] Key type .llcrypt registered [ 1308.347417] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 04:58:50 (1769075930) [ 1308.889276] Key type .llcrypt unregistered [ 1308.892264] Key type ._llcrypt unregistered [ 1315.199901] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1315.378732] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1315.405735] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1315.412854] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1318.079840] Key type ._llcrypt registered [ 1318.083434] Key type .llcrypt registered [ 1318.291370] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1318.313294] alg: No test for adler32 (adler32-zlib) [ 1319.253708] LNet: 62656:0:(config.c:1575:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1319.263803] LNet: Added LNI 192.168.201.50@tcp [8/256/0/180] [ 1319.312258] LNet: 62670:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1319.342464] LNetError: 62670:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1319.371683] LNet: Removed LNI 192.168.201.50@tcp [ 1320.018646] Key type .llcrypt unregistered [ 1320.022825] Key type ._llcrypt unregistered [ 1326.065471] Key type ._llcrypt registered [ 1326.067669] Key type .llcrypt registered [ 1326.315028] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1326.343945] alg: No test for adler32 (adler32-zlib) [ 1327.402164] LNet: 62925:0:(config.c:1575:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1327.426090] LNet: Added LNI 192.168.201.50@tcp [8/256/0/180] [ 1327.436146] LNet: Added LNI 10.1.2.3@tcp1 [8/256/0/180] [ 1327.524169] LNet: 62939:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1327.539391] LNetError: 62939:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1327.548683] LNet: Removed LNI 192.168.201.50@tcp [ 1327.571808] LNet: Removed LNI 10.1.2.3@tcp1 [ 1328.479550] Key type .llcrypt unregistered [ 1328.482658] Key type ._llcrypt unregistered [ 1335.729523] Key type ._llcrypt registered [ 1335.738972] Key type .llcrypt registered [ 1335.990337] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1336.028873] alg: No test for adler32 (adler32-zlib) [ 1336.987507] LNet: 63195:0:(config.c:1575:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1336.999795] LNet: Added LNI 192.168.201.50@tcp [8/256/0/180] [ 1337.003440] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1337.040545] LNet: 63209:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1337.046175] LNetError: 63209:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1337.049570] LNet: Removed LNI 192.168.201.50@tcp [ 1337.061602] LNet: Removed LNI 10.1.2.3@tcp [ 1337.783468] Key type .llcrypt unregistered [ 1337.785941] Key type ._llcrypt unregistered [ 1344.663665] Key type ._llcrypt registered [ 1344.667506] Key type .llcrypt registered [ 1344.900477] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1344.933387] alg: No test for adler32 (adler32-zlib) [ 1345.995723] LNet: 63466:0:(config.c:1575:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1346.033870] LNet: Added LNI 192.168.201.50@tcp [8/256/0/180] [ 1346.039550] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1346.121184] LNet: 63480:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1346.134343] LNetError: 63480:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1346.149541] LNet: Removed LNI 192.168.201.50@tcp [ 1346.198882] LNet: Removed LNI 10.1.2.3@tcp [ 1347.292399] Key type .llcrypt unregistered [ 1347.295885] Key type ._llcrypt unregistered [ 1354.194572] Key type ._llcrypt registered [ 1354.196954] Key type .llcrypt registered [ 1354.371728] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1354.394092] alg: No test for adler32 (adler32-zlib) [ 1355.262478] LNet: 63741:0:(config.c:1575:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1355.274573] LNet: Added LNI 192.168.201.50@tcp [8/256/0/180] [ 1355.277519] LNetError: 63741:0:(config.c:1560:lnet_inet_select()) ksocklnd: failed to find UP interface test1pg [ 1355.282513] LNetError: Error -100 starting up LNI tcp [ 1355.296763] LNet: Removed LNI 192.168.201.50@tcp [ 1355.892500] Key type .llcrypt unregistered [ 1355.899531] Key type ._llcrypt unregistered [ 1370.622904] Key type ._llcrypt registered [ 1370.627254] Key type .llcrypt registered [ 1371.861304] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 04:59:53 (1769075993) [ 1372.959647] Key type .llcrypt unregistered [ 1372.963864] Key type ._llcrypt unregistered [ 1380.194723] Key type ._llcrypt registered [ 1380.201415] Key type .llcrypt registered [ 1380.420749] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1380.451219] alg: No test for adler32 (adler32-zlib) [ 1381.417755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1381.447469] LNet: Added LNI 192.168.201.50@tcp [8/256/0/180] [ 1391.362495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.150@tcp [ 1391.980659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.50@tcp -d 192.168.201.150@tcp -m GET -e network_timeout -r 1 [ 1392.021186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.50@tcp -d 192.168.201.50@tcp -m GET -e network_timeout -r 1 [ 1392.075958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.150@tcp [ 1392.117272] LNet: There was an unexpected network error while writing to 192.168.201.150: rc = -22 [ 1392.363265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1392.389570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1392.417993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1392.447681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1392.472559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1392.497328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1392.527954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1392.568414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1392.590838] LNet: 64994:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1392.611302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1392.654810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1392.701245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1392.731678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1392.736613] LNet: 1 local NIs in recovery (showing 1): 192.168.201.50@tcp [ 1392.781678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1392.817490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1393.087308] LNet: 65032:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1393.092854] LNetError: 65032:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1393.113605] LNet: Removed LNI 192.168.201.50@tcp [ 1393.469490] Key type .llcrypt unregistered [ 1393.470989] Key type ._llcrypt unregistered [ 1413.476543] Key type ._llcrypt registered [ 1413.479504] Key type .llcrypt registered [ 1413.642483] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1413.659541] alg: No test for adler32 (adler32-zlib) [ 1414.553189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1414.583695] LNet: Added LNI 192.168.201.50@tcp [8/256/0/180] [ 1421.948745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.150@tcp [ 1429.271803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1429.323956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1429.349657] LNet: Added LNI 192.168.201.50@tcp1 [8/256/0/180] [ 1430.627247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.50@tcp -d 192.168.201.150@tcp -m GET -e network_timeout -r 1 [ 1430.658365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.50@tcp -d 192.168.201.150@tcp1 -m GET -e network_timeout -r 1 [ 1430.689213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.50@tcp -d 192.168.201.50@tcp -m GET -e network_timeout -r 1 [ 1430.727353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.50@tcp -d 192.168.201.50@tcp1 -m GET -e network_timeout -r 1 [ 1430.757553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.50@tcp1 -d 192.168.201.150@tcp -m GET -e network_timeout -r 1 [ 1430.792181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.50@tcp1 -d 192.168.201.150@tcp1 -m GET -e network_timeout -r 1 [ 1430.829770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.50@tcp1 -d 192.168.201.50@tcp -m GET -e network_timeout -r 1 [ 1430.871796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.50@tcp1 -d 192.168.201.50@tcp1 -m GET -e network_timeout -r 1 [ 1430.928683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.150@tcp [ 1430.947336] LNet: There was an unexpected network error while writing to 192.168.201.150: rc = -22 [ 1431.007348] LNet: 1 local NIs in recovery (showing 1): 192.168.201.50@tcp [ 1431.014217] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.150@tcp [ 1431.190719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1431.233460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1431.270428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1431.304266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1431.343687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1431.381125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1431.411475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1431.441590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1431.456715] LNet: 65906:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1431.470392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1431.494398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1431.523811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1431.548282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1431.573529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1431.610102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1436.010444] LNet: There was an unexpected network error while writing to 192.168.201.150: rc = -22 [ 1436.024820] LNet: Skipped 2 previous similar messages [ 1436.563223] LNet: 65979:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1436.574322] LNetError: 65979:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1436.585857] LNet: Removed LNI 192.168.201.50@tcp [ 1436.597565] LNet: Removed LNI 192.168.201.50@tcp1 [ 1437.074790] Key type .llcrypt unregistered [ 1437.084445] Key type ._llcrypt unregistered [ 1454.280780] Key type ._llcrypt registered [ 1454.283253] Key type .llcrypt registered [ 1455.512823] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 05:01:17 (1769076077) [ 1455.882235] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1455.892763] alg: No test for adler32 (adler32-zlib) [ 1456.878091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1456.947465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1456.982604] LNet: Added LNI 192.168.201.50@tcp [8/256/0/180] [ 1457.656607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1457.677546] LNet: Added LNI 192.168.201.50@tcp1 [8/256/0/180] [ 1458.532058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.50@tcp [ 1458.592389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1458.628300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1458.661957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1458.714372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1458.779482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1458.809871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.201.50@tcp [ 1458.830751] LNetError: 67091:0:(api-ni.c:4348:lnet_ni_set_healthv()) manually adding local NI 192.168.201.50@tcp to recovery [ 1458.975353] LNet: 1 local NIs in recovery (showing 1): 192.168.201.50@tcp [ 1458.977799] LNet: There was an unexpected network error while writing to 192.168.201.50: rc = -22 [ 1460.013467] LNet: There was an unexpected network error while writing to 192.168.201.50: rc = -22 [ 1460.173632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1464.095337] LNet: There was an unexpected network error while writing to 192.168.201.50: rc = -22 [ 1465.050854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1465.110436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1465.249535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1465.293738] LNet: 67188:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1465.298463] LNetError: 67188:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1465.304367] LNet: Removed LNI 192.168.201.50@tcp [ 1465.315752] LNet: Removed LNI 192.168.201.50@tcp1 [ 1465.348068] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1465.402968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1465.429496] LNet: Added LNI 192.168.201.50@tcp [8/256/0/180] [ 1466.216391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1466.254428] LNet: Added LNI 192.168.201.50@tcp1 [8/256/0/180] [ 1467.044322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.50@tcp [ 1467.079329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1467.134532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1467.169729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1467.196238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1467.221994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1467.254132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1467.290838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.201.50@tcp [ 1467.310415] LNetError: 67461:0:(api-ni.c:4348:lnet_ni_set_healthv()) manually adding local NI 192.168.201.50@tcp to recovery [ 1467.425762] LNet: There was an unexpected network error while writing to 192.168.201.50: rc = -22 [ 1469.744628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1473.569097] LNet: There was an unexpected network error while writing to 192.168.201.50: rc = -22 [ 1473.584850] LNet: Skipped 1 previous similar message [ 1478.057717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1478.116452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1478.172343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1485.728429] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 05:01:47 (1769076107) [ 1485.968528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1485.999584] LNet: 68168:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1486.006660] LNetError: 68168:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1486.018177] LNetError: 68168:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1486.026829] LNet: Removed LNI 192.168.201.50@tcp [ 1486.043850] LNet: Removed LNI 192.168.201.50@tcp1 [ 1486.121888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1486.217876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1486.238858] LNet: Added LNI 192.168.201.50@tcp [8/256/0/180] [ 1487.107303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1487.140244] LNet: Added LNI 192.168.201.50@tcp1 [8/256/0/180] [ 1488.210653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.50@tcp [ 1488.293959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1488.342521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1488.376619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1488.417720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1488.453141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1488.493158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.201.50@tcp --health 0 [ 1488.597919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1489.247578] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.50@tcp [ 1489.248601] LNet: There was an unexpected network error while writing to 192.168.201.50: rc = -22 [ 1489.258858] LNet: Skipped 1 previous similar message [ 1504.238517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1504.394711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1504.438115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1504.497507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.201.50@tcp --health 500 [ 1504.592339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1511.308467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1511.376646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1511.406489] LNet: 68660:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1511.411349] LNetError: 68660:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1511.418254] LNetError: 68660:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1511.424566] LNet: Removed LNI 192.168.201.50@tcp [ 1511.445647] LNet: Removed LNI 192.168.201.50@tcp1 [ 1511.475823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1511.529744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1511.551932] LNet: Added LNI 192.168.201.50@tcp [8/256/0/180] [ 1512.185849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1512.211333] LNet: Added LNI 192.168.201.50@tcp1 [8/256/0/180] [ 1512.901624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.50@tcp [ 1512.964647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1513.053374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1513.105533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1513.168641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1513.243990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1513.289360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1513.315759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.201.50@tcp --health 0 [ 1514.593600] LNet: There was an unexpected network error while writing to 192.168.201.50: rc = -22 [ 1514.604358] LNet: Skipped 3 previous similar messages [ 1515.754367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1526.068423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1526.131692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1526.193945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1534.477984] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 05:02:36 (1769076156) [ 1535.007329] LNet: 69717:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1535.018145] LNetError: 69717:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1535.027325] LNetError: 69717:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1535.042993] LNet: Removed LNI 192.168.201.50@tcp [ 1535.062440] LNet: Removed LNI 192.168.201.50@tcp1 [ 1535.564606] Key type .llcrypt unregistered [ 1535.566916] Key type ._llcrypt unregistered [ 1541.558768] Key type ._llcrypt registered [ 1541.560817] Key type .llcrypt registered [ 1541.830853] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1541.861129] alg: No test for adler32 (adler32-zlib) [ 1542.815647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1542.850703] LNet: Added LNI 192.168.201.50@tcp [8/256/0/180] [ 1551.981752] Lustre: DEBUG MARKER: Initial discovery [ 1552.386071] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.201.150@tcp [ 1554.714380] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1554.920161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1554.946690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.201.150@tcp [ 1554.963782] LNet: There was an unexpected network error while writing to 192.168.201.150: rc = -22 [ 1554.989513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1555.044970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.201.150@tcp -r 1 -m GET -l 3 [ 1555.103984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.201.150@tcp -r 1 -m GET -e local_error [ 1555.170338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.201.150@tcp -r 1 -m PUT -l 6 [ 1555.172605] LNet: 1 local NIs in recovery (showing 1): 192.168.201.50@tcp [ 1556.757509] Lustre: DEBUG MARKER: Force oleg150-client.virtnet to discover 192.168.201.150@tcp (in background) [ 1556.855489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.201.150@tcp [ 1558.245869] Lustre: DEBUG MARKER: Force oleg150-server to discover 192.168.201.50@tcp [ 1559.909128] LNet: There was an unexpected network error while writing to 192.168.201.150: rc = -22 [ 1560.860664] Lustre: DEBUG MARKER: Wait for 70467 [ 1562.425470] Lustre: DEBUG MARKER: Finished wait on 70467 [ 1562.658943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay del -a [ 1562.714802] LNet: There was an unexpected network error while writing to 192.168.201.150: rc = -22 [ 1562.741985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1563.062682] LNet: 70924:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1563.069587] LNetError: 70924:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1564.133212] LNet: Removed LNI 192.168.201.50@tcp [ 1564.682353] Key type .llcrypt unregistered [ 1564.693472] Key type ._llcrypt unregistered [ 1581.834436] Key type ._llcrypt registered [ 1581.839163] Key type .llcrypt registered [ 1583.374427] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 05:03:24 (1769076204) [ 1584.256367] Key type .llcrypt unregistered [ 1584.258151] Key type ._llcrypt unregistered [ 1590.784176] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1591.126043] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1591.150942] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1591.168389] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1593.990102] Key type ._llcrypt registered [ 1593.996657] Key type .llcrypt registered [ 1594.266879] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1594.298340] alg: No test for adler32 (adler32-zlib) [ 1595.280045] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1595.348226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1595.389742] LNet: Added LNI 192.168.201.50@tcp [8/256/0/180] [ 1596.353704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1596.388279] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1605.303981] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 05:03:47 (1769076227) [ 1605.680614] LNet: 72998:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1605.689752] LNetError: 72998:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1605.695529] LNet: Removed LNI 192.168.201.50@tcp [ 1605.708888] LNet: Removed LNI 10.1.2.3@tcp [ 1606.693400] Key type .llcrypt unregistered [ 1606.695469] Key type ._llcrypt unregistered [ 1613.497279] Key type ._llcrypt registered [ 1613.499556] Key type .llcrypt registered [ 1613.720684] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1613.742757] alg: No test for adler32 (adler32-zlib) [ 1614.742751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1614.822694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1614.848415] LNet: Added LNI 192.168.201.50@tcp [8/256/0/180] [ 1615.992695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1616.017050] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1617.351364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.201.50@tcp [ 1618.464319] LNetError: Unexpected error -22 connecting to 192.168.201.50@tcp at host 192.168.201.50:988 [ 1628.052635] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 05:04:08 (1769076248) [ 1628.498316] LNet: 74126:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1628.522342] LNetError: 74126:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1628.532456] LNet: Removed LNI 192.168.201.50@tcp [ 1628.549181] LNet: Removed LNI 10.1.2.3@tcp [ 1629.893679] Key type .llcrypt unregistered [ 1629.898757] Key type ._llcrypt unregistered [ 1639.904964] Key type ._llcrypt registered [ 1639.906923] Key type .llcrypt registered [ 1640.115868] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1640.143324] alg: No test for adler32 (adler32-zlib) [ 1641.028349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1641.114770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1641.153850] LNet: Added LNI 192.168.201.50@tcp1 [8/256/0/180] [ 1642.294260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1642.339042] LNet: Added LNI 192.168.201.50@tcp2 [8/256/0/180] [ 1643.587304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.201.50@tcp1 --nid 192.168.201.50@tcp2 [ 1649.066892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.50@tcp1 192.168.201.50@tcp2 [ 1649.109155] LNetError: 74959:0:(lib-move.c:2264:lnet_handle_find_routed_path()) Unable to send message from 192.168.201.50@tcp1 to 192.168.201.50@tcp2 - Route table may be misconfigured [ 1649.146639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.50@tcp2 192.168.201.50@tcp1 [ 1658.291802] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 05:04:40 (1769076280) [ 1658.376809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1658.400076] LNet: 75525:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1658.408772] LNetError: 75525:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1658.419981] LNet: Removed LNI 192.168.201.50@tcp1 [ 1659.494287] LNet: Removed LNI 192.168.201.50@tcp2 [ 1659.545419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1659.610768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1659.645588] LNet: Added LNI 192.168.201.50@tcp [8/256/0/180] [ 1660.422895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1660.439496] LNet: Added LNI 192.168.201.50@tcp1 [8/256/0/180] [ 1661.063637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.50@tcp [ 1661.101493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.50@tcp 192.168.201.50@tcp [ 1661.133243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.50@tcp1 192.168.201.50@tcp1 [ 1661.211331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.201.50@tcp -d 192.168.201.50@tcp -e network_timeout [ 1661.263323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.201.50@tcp -d 192.168.201.50@tcp1 -e network_timeout [ 1661.288840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.201.50@tcp1 -d 192.168.201.50@tcp -e network_timeout [ 1661.327144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.201.50@tcp1 -d 192.168.201.50@tcp1 -e network_timeout [ 1661.371569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.50@tcp [ 1661.391770] LNet: There was an unexpected network error while writing to 192.168.201.50: rc = -22 [ 1661.408974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1661.498794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1661.587273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1661.665969] LNet: 1 local NIs in recovery (showing 1): 192.168.201.50@tcp [ 1667.380211] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 05:04:49 (1769076289) [ 1667.588519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1667.628225] LNet: 76395:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1667.654779] LNetError: 76395:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1667.665149] LNetError: 76395:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1667.678297] LNet: Removed LNI 192.168.201.50@tcp [ 1668.717825] LNet: Removed LNI 192.168.201.50@tcp1 [ 1668.771991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1668.888139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1669.126826] LNet: 76431:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1669.544228] Key type .llcrypt unregistered [ 1669.546359] Key type ._llcrypt unregistered [ 1681.677657] Key type ._llcrypt registered [ 1681.683617] Key type .llcrypt registered [ 1683.224831] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 05:05:04 (1769076304) [ 1683.553421] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1683.568380] alg: No test for adler32 (adler32-zlib) [ 1684.477551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1685.802312] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1687.442951] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 05:05:09 (1769076309) [ 1687.595508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1687.613496] LNet: 77526:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1687.658507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1687.699939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1687.716911] LNet: Added LNI 192.168.201.50@tcp [8/256/0/180] [ 1688.551823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1688.584908] LNet: Added LNI 192.168.201.50@tcp1 [8/256/0/180] [ 1689.724914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.50@tcp [ 1689.758982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.50@tcp1 [ 1689.797987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.50@tcp1 [ 1696.817846] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 05:05:18 (1769076318) [ 1698.928118] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg150-server [ 1700.774709] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 05:05:22 (1769076322) [ 1703.212597] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg150-server [ 1704.892434] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 05:05:26 (1769076326) [ 1706.937743] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg150-server [ 1709.499157] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 05:05:30 (1769076330) [ 1711.337553] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg150-server [ 1714.072665] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 05:05:35 (1769076335) [ 1716.447977] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg150-server [ 1718.936389] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 05:05:40 (1769076340) [ 1721.160918] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg150-server [ 1723.191174] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 05:05:44 (1769076344) [ 1725.147259] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg150-server [ 1727.159768] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 05:05:48 (1769076348) [ 1729.440991] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg150-server [ 1731.746964] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 05:05:53 (1769076353) [ 1734.184848] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg150-server [ 1736.570142] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 05:05:58 (1769076358) [ 1737.119787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1737.161043] LNet: 81408:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1737.191610] LNetError: 81408:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1737.203952] LNet: Removed LNI 192.168.201.50@tcp [ 1738.226493] LNet: Removed LNI 192.168.201.50@tcp1 [ 1738.254044] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1738.317273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1738.355855] LNet: Added LNI 192.168.201.50@tcp [8/256/0/180] [ 1739.254838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1739.359679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.50@tcp [ 1739.671798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1739.696689] LNet: 81555:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1739.727982] LNetError: 81555:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1739.730883] LNetError: 81555:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1739.745534] LNet: Removed LNI 192.168.201.50@tcp [ 1739.780516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1739.881621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1739.924735] LNet: Added LNI 192.168.201.50@tcp [8/256/0/180] [ 1740.891839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.50@tcp [ 1741.109414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1741.146824] LNetError: 81700:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1741.166364] LNet: Removed LNI 192.168.201.50@tcp [ 1741.212364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1741.291814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1741.338628] LNet: Added LNI 192.168.201.50@tcp [8/256/0/180] [ 1742.335800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1742.457942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.50@tcp [ 1742.717670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1742.733732] LNet: 81848:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1742.737958] LNet: 81848:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1743.790618] LNet: Removed LNI 192.168.201.50@tcp [ 1743.874143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1743.972513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1744.014494] LNet: Added LNI 192.168.201.50@tcp [8/256/0/180] [ 1745.152864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.50@tcp [ 1745.463937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1745.522825] LNetError: 81993:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1745.531085] LNetError: 81993:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1746.603228] LNet: Removed LNI 192.168.201.50@tcp [ 1746.666305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1746.798309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1746.838412] LNet: Added LNI 192.168.201.50@tcp [8/256/0/180] [ 1748.456168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1748.580696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.50@tcp [ 1749.011180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1749.084598] LNet: 82141:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1749.097178] LNet: 82141:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1750.181704] LNet: Removed LNI 192.168.201.50@tcp [ 1750.203930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1750.307592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1750.357186] LNet: Added LNI 192.168.201.50@tcp [8/256/0/180] [ 1751.575772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.50@tcp [ 1751.760816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1751.792582] LNetError: 82286:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1751.797935] LNetError: 82286:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1752.871594] LNet: Removed LNI 192.168.201.50@tcp [ 1752.918918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1753.080927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1753.164903] LNet: Added LNI 192.168.201.50@tcp [8/256/0/180] [ 1754.696335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1754.823350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.50@tcp [ 1755.203325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1756.264322] LNet: Removed LNI 192.168.201.50@tcp [ 1756.318221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1756.479842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1756.526354] LNet: Added LNI 192.168.201.50@tcp [8/256/0/180] [ 1757.767227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.50@tcp [ 1758.137901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1758.179690] LNet: 82579:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1758.186398] LNet: 82579:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 1759.213604] LNet: Removed LNI 192.168.201.50@tcp [ 1759.260942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1759.321929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1759.345670] LNet: Added LNI 192.168.201.50@tcp [8/256/0/180] [ 1760.033908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1760.153714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.50@tcp [ 1760.499557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1760.572518] LNetError: 82726:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1760.576461] LNetError: 82726:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1760.616069] LNet: Removed LNI 192.168.201.50@tcp [ 1760.659171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1760.774778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1760.808720] LNet: Added LNI 192.168.201.50@tcp [8/256/0/180] [ 1761.688327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.50@tcp [ 1762.070230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1763.175898] LNet: Removed LNI 192.168.201.50@tcp [ 1763.253782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1763.309754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1763.333895] LNet: Added LNI 192.168.201.50@tcp [8/256/0/180] [ 1764.036688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1764.105261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.50@tcp [ 1764.292524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1765.351296] LNet: Removed LNI 192.168.201.50@tcp [ 1765.398032] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1765.474396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1765.518647] LNet: Added LNI 192.168.201.50@tcp [8/256/0/180] [ 1766.456175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.50@tcp [ 1766.644524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1767.728234] LNet: Removed LNI 192.168.201.50@tcp [ 1767.782701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1767.858538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1767.888315] LNet: Added LNI 192.168.201.50@tcp [8/256/0/180] [ 1768.662785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1768.713806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1768.736277] LNet: Removed LNI 192.168.201.50@tcp [ 1768.761273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1768.861748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1768.907130] LNet: Added LNI 192.168.201.50@tcp [8/256/0/180] [ 1770.018456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1777.908968] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 05:06:39 (1769076399) [ 1778.130066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1778.190829] LNet: 83996:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1778.194299] LNet: 83996:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 5 previous similar messages [ 1778.238131] LNetError: 83996:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1778.255983] LNetError: 83996:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 1778.300216] LNet: Removed LNI 192.168.201.50@tcp [ 1778.351785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1778.470522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1778.557179] LNet: Added LNI 192.168.201.50@tcp231 [8/256/0/180] [ 1780.261861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1780.310694] LNet: Removed LNI 192.168.201.50@tcp231 [ 1780.327729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1780.426984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1780.473503] LNet: Added LNI 192.168.201.50@tcp231 [8/256/0/180] [ 1780.502277] LNet: 84148:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1781.284465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1781.342961] LNet: Removed LNI 192.168.201.50@tcp231 [ 1781.399213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1781.456925] LNet: Added LNI 192.168.201.50@tcp231 [8/256/0/180] [ 1782.754948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1782.799122] LNet: Removed LNI 192.168.201.50@tcp231 [ 1782.819708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1782.934595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1782.979629] LNet: Added LNI 192.168.201.50@tcp231 [8/256/0/0] [ 1782.984942] LNet: 84300:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1783.484727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1783.531357] LNet: Removed LNI 192.168.201.50@tcp231 [ 1783.553720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1783.606600] LNet: Added LNI 192.168.201.50@tcp231 [8/256/0/0] [ 1785.195711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1785.259861] LNet: Removed LNI 192.168.201.50@tcp231 [ 1785.287292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1785.346334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1785.366109] LNet: Added LNI 192.168.201.50@tcp231 [8/256/0/60] [ 1785.371745] LNet: 84452:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1785.933833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1785.959132] LNet: Removed LNI 192.168.201.50@tcp231 [ 1785.969461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1786.022519] LNet: Added LNI 192.168.201.50@tcp231 [8/256/0/60] [ 1787.596304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1787.642060] LNet: Removed LNI 192.168.201.50@tcp231 [ 1787.680631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1787.804607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1787.862747] LNet: Added LNI 192.168.201.50@tcp231 [8/256/0/180] [ 1787.876957] LNet: 84603:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1788.704552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1788.754914] LNet: Removed LNI 192.168.201.50@tcp231 [ 1788.786076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1788.822212] LNet: Added LNI 192.168.201.50@tcp231 [8/256/0/180] [ 1800.951863] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 05:07:02 (1769076422) [ 1801.176690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1801.260690] LNet: Removed LNI 192.168.201.50@tcp231 [ 1801.309994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1801.381630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1801.396534] LNet: Added LNI 192.168.201.50@tcp232 [8/256/0/180] [ 1802.234680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1802.297462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1802.354378] LNet: Removed LNI 192.168.201.50@tcp232 [ 1802.373563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1802.449912] LNet: Added LNI 192.168.201.50@tcp232 [8/256/0/180] [ 1803.580760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1803.645957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1803.693535] LNet: Removed LNI 192.168.201.50@tcp232 [ 1803.729954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1803.787606] LNet: Added LNI 192.168.201.50@tcp232 [8/256/0/180] [ 1804.920663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1804.970039] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1805.019340] LNet: Removed LNI 192.168.201.50@tcp232 [ 1805.064555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1805.103606] LNet: Added LNI 192.168.201.50@tcp232 [8/256/0/180] [ 1806.210722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1806.248911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1806.279888] LNet: Removed LNI 192.168.201.50@tcp232 [ 1806.302347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1813.875671] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 05:07:15 (1769076435) [ 1814.532665] LNet: 86443:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1814.535944] LNet: 86443:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 5 previous similar messages [ 1815.354635] Key type .llcrypt unregistered [ 1815.360204] Key type ._llcrypt unregistered [ 1823.071612] Key type ._llcrypt registered [ 1823.077148] Key type .llcrypt registered [ 1823.329796] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1823.355781] alg: No test for adler32 (adler32-zlib) [ 1824.275433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1824.327231] LNet: Added LNI 192.168.201.50@tcp [8/256/0/180] [ 1832.724937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.150@tcp [ 1843.720756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1843.780357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1843.826525] LNet: Added LNI 192.168.201.50@tcp1 [8/256/0/180] [ 1845.551518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.50@tcp -d 192.168.201.150@tcp -m GET -e local_interrupt -r 2 [ 1845.600408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.50@tcp -d 192.168.201.150@tcp1 -m GET -e local_interrupt -r 2 [ 1845.653885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.50@tcp -d 192.168.201.50@tcp -m GET -e local_interrupt -r 2 [ 1845.718570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.50@tcp -d 192.168.201.50@tcp1 -m GET -e local_interrupt -r 2 [ 1845.762607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.50@tcp1 -d 192.168.201.150@tcp -m GET -e local_interrupt -r 2 [ 1845.850738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.50@tcp1 -d 192.168.201.150@tcp1 -m GET -e local_interrupt -r 2 [ 1845.901173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.50@tcp1 -d 192.168.201.50@tcp -m GET -e local_interrupt -r 2 [ 1845.984116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.50@tcp1 -d 192.168.201.50@tcp1 -m GET -e local_interrupt -r 2 [ 1846.068696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.50@tcp 192.168.201.150@tcp [ 1846.107782] LNet: There was an unexpected network error while writing to 192.168.201.150: rc = -22 [ 1846.117889] LNet: 1 local NIs in recovery (showing 1): 192.168.201.50@tcp [ 1846.417678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1846.443127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1846.467816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1846.520584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1846.578743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1846.624549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1846.667774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1846.705912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1846.735744] LNet: 87103:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1846.750992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1846.786620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1846.813150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1846.836625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1846.869695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1846.909987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1847.566274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.50@tcp -d 192.168.201.150@tcp -m GET -e local_dropped -r 2 [ 1847.650791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.50@tcp -d 192.168.201.150@tcp1 -m GET -e local_dropped -r 2 [ 1847.721675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.50@tcp -d 192.168.201.50@tcp -m GET -e local_dropped -r 2 [ 1847.791690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.50@tcp -d 192.168.201.50@tcp1 -m GET -e local_dropped -r 2 [ 1847.845734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.50@tcp1 -d 192.168.201.150@tcp -m GET -e local_dropped -r 2 [ 1847.951023] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.50@tcp1 -d 192.168.201.150@tcp1 -m GET -e local_dropped -r 2 [ 1848.043125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.50@tcp1 -d 192.168.201.50@tcp -m GET -e local_dropped -r 2 [ 1848.113377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.50@tcp1 -d 192.168.201.50@tcp1 -m GET -e local_dropped -r 2 [ 1848.178372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.50@tcp 192.168.201.150@tcp [ 1848.205814] LNet: There was an unexpected network error while writing to 192.168.201.150: rc = -22 [ 1848.490774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1848.519992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1848.559699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1848.604346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1848.636648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1848.669765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1848.717840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1848.764493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1848.807758] LNet: 87223:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1848.830231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1848.888254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1848.951959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1849.015209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1849.081312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1849.159883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1849.933185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.50@tcp -d 192.168.201.150@tcp -m GET -e local_aborted -r 2 [ 1849.991308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.50@tcp -d 192.168.201.150@tcp1 -m GET -e local_aborted -r 2 [ 1850.044713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.50@tcp -d 192.168.201.50@tcp -m GET -e local_aborted -r 2 [ 1850.082954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.50@tcp -d 192.168.201.50@tcp1 -m GET -e local_aborted -r 2 [ 1850.126919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.50@tcp1 -d 192.168.201.150@tcp -m GET -e local_aborted -r 2 [ 1850.185784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.50@tcp1 -d 192.168.201.150@tcp1 -m GET -e local_aborted -r 2 [ 1850.251576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.50@tcp1 -d 192.168.201.50@tcp -m GET -e local_aborted -r 2 [ 1850.281822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.50@tcp1 -d 192.168.201.50@tcp1 -m GET -e local_aborted -r 2 [ 1850.333436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.50@tcp 192.168.201.150@tcp [ 1850.362064] LNet: There was an unexpected network error while writing to 192.168.201.150: rc = -22 [ 1850.720204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1850.751952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1850.787284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1850.816764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1850.851853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1850.886731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1850.924361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1850.966185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1850.992129] LNet: 87344:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1851.030187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1851.065971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1851.100557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1851.151082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1851.187786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1851.216592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1851.884669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.50@tcp -d 192.168.201.150@tcp -m GET -e local_no_route -r 2 [ 1851.922832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.50@tcp -d 192.168.201.150@tcp1 -m GET -e local_no_route -r 2 [ 1851.956761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.50@tcp -d 192.168.201.50@tcp -m GET -e local_no_route -r 2 [ 1852.019862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.50@tcp -d 192.168.201.50@tcp1 -m GET -e local_no_route -r 2 [ 1852.059294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.50@tcp1 -d 192.168.201.150@tcp -m GET -e local_no_route -r 2 [ 1852.094435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.50@tcp1 -d 192.168.201.150@tcp1 -m GET -e local_no_route -r 2 [ 1852.142327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.50@tcp1 -d 192.168.201.50@tcp -m GET -e local_no_route -r 2 [ 1852.194567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.50@tcp1 -d 192.168.201.50@tcp1 -m GET -e local_no_route -r 2 [ 1852.235305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.50@tcp 192.168.201.150@tcp [ 1852.631953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1852.680982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1852.745760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1852.817789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1852.862085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1852.887350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1852.922255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1852.974846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1853.012872] LNet: 87464:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1853.044223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1853.081096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1853.120698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1853.157767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1853.203952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1853.249490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1853.894350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.50@tcp -d 192.168.201.150@tcp -m GET -e local_timeout -r 2 [ 1853.971682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.50@tcp -d 192.168.201.150@tcp1 -m GET -e local_timeout -r 2 [ 1854.030175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.50@tcp -d 192.168.201.50@tcp -m GET -e local_timeout -r 2 [ 1854.082955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.50@tcp -d 192.168.201.50@tcp1 -m GET -e local_timeout -r 2 [ 1854.146242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.50@tcp1 -d 192.168.201.150@tcp -m GET -e local_timeout -r 2 [ 1854.198386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.50@tcp1 -d 192.168.201.150@tcp1 -m GET -e local_timeout -r 2 [ 1854.258962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.50@tcp1 -d 192.168.201.50@tcp -m GET -e local_timeout -r 2 [ 1854.338295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.50@tcp1 -d 192.168.201.50@tcp1 -m GET -e local_timeout -r 2 [ 1854.410525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.50@tcp 192.168.201.150@tcp [ 1854.438867] LNet: There was an unexpected network error while writing to 192.168.201.150: rc = -22 [ 1854.441836] LNet: Skipped 1 previous similar message [ 1854.858849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1854.887390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1854.929835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1855.008418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1855.065087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1855.105406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1855.167957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1855.235861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1855.275830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1855.326222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1855.388302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1855.483883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1855.556969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1855.628577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1856.332390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.50@tcp -d 192.168.201.150@tcp -m GET -e remote_dropped -r 2 [ 1856.362979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.50@tcp -d 192.168.201.150@tcp1 -m GET -e remote_dropped -r 2 [ 1856.393819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.50@tcp -d 192.168.201.50@tcp -m GET -e remote_dropped -r 2 [ 1856.430948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.50@tcp -d 192.168.201.50@tcp1 -m GET -e remote_dropped -r 2 [ 1856.464861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.50@tcp1 -d 192.168.201.150@tcp -m GET -e remote_dropped -r 2 [ 1856.511174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.50@tcp1 -d 192.168.201.150@tcp1 -m GET -e remote_dropped -r 2 [ 1856.558712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.50@tcp1 -d 192.168.201.50@tcp -m GET -e remote_dropped -r 2 [ 1856.612975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.50@tcp1 -d 192.168.201.50@tcp1 -m GET -e remote_dropped -r 2 [ 1856.665317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.50@tcp 192.168.201.150@tcp [ 1856.704732] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.150@tcp [ 1857.015730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1857.048292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1857.081363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1857.118460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1857.157655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1857.197481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1857.229952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1857.275258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1857.318756] LNet: 87704:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1857.333850] LNet: 87704:0:(api-ni.c:343:recovery_interval_set()) Skipped 1 previous similar message [ 1857.374763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1857.434105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1857.477966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1857.511885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1857.559573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1857.622596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1864.017634] LNet: 87778:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1864.021615] LNetError: 87778:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1864.027810] LNet: Removed LNI 192.168.201.50@tcp [ 1864.035514] LNet: Removed LNI 192.168.201.50@tcp1 [ 1864.432767] Key type .llcrypt unregistered [ 1864.442682] Key type ._llcrypt unregistered [ 1883.434754] Key type ._llcrypt registered [ 1883.441269] Key type .llcrypt registered [ 1884.884855] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 05:08:26 (1769076506) [ 1885.266227] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1885.279852] alg: No test for adler32 (adler32-zlib) [ 1886.277375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1886.375316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1886.418886] LNet: Added LNI 192.168.201.50@tcp [8/256/0/180] [ 1887.761943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1887.812249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.201.50@tcp [ 1887.843191] LNetError: 88764:0:(api-ni.c:4348:lnet_ni_set_healthv()) manually adding local NI 192.168.201.50@tcp to recovery [ 1887.977762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.201.50@tcp [ 1888.126658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1896.483069] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 05:08:37 (1769076517) [ 1896.829489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1896.848890] LNet: 89354:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1896.854723] LNetError: 89354:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1896.867705] LNet: Removed LNI 192.168.201.50@tcp [ 1896.879339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1896.919558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1896.937737] LNet: Added LNI 192.168.201.50@tcp [8/256/0/180] [ 1897.556203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1897.584357] LNet: Added LNI 192.168.201.50@tcp1 [8/256/0/180] [ 1898.886488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.201.50@tcp [ 1898.920925] LNetError: 89620:0:(api-ni.c:4348:lnet_ni_set_healthv()) manually adding local NI 192.168.201.50@tcp to recovery [ 1898.980089] LNet: 1 local NIs in recovery (showing 1): 192.168.201.50@tcp [ 1898.986332] LNet: There was an unexpected network error while writing to 192.168.201.50: rc = -22 [ 1899.181903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.201.50@tcp [ 1909.485042] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 05:08:51 (1769076531) [ 1911.656900] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg150-server [ 1913.448139] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 05:08:55 (1769076535) [ 1913.795885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1913.830212] LNet: 90568:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1913.853034] LNetError: 90568:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1913.864119] LNet: Removed LNI 192.168.201.50@tcp [ 1913.880558] LNet: Removed LNI 192.168.201.50@tcp1 [ 1913.927305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1913.977578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1913.995323] LNet: Added LNI 192.168.201.50@tcp [8/256/0/180] [ 1914.622166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1914.681826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1914.725818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1914.791714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1914.832932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1914.867663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1923.259035] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 05:09:04 (1769076544) [ 1923.478307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1923.508765] LNet: 91290:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1923.516111] LNetError: 91290:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1923.526915] LNetError: 91290:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1923.568603] LNet: Removed LNI 192.168.201.50@tcp [ 1923.653396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1923.704369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1923.723302] LNet: Added LNI 192.168.201.50@tcp [8/256/0/180] [ 1931.182701] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 05:09:13 (1769076553) [ 1931.308982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1931.366785] LNet: 91991:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1931.374715] LNetError: 91991:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1931.398529] LNet: Removed LNI 192.168.201.50@tcp [ 1931.421986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1931.501768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1931.527902] LNet: Added LNI 192.168.201.50@tcp [8/256/0/180] [ 1940.093591] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 05:09:21 (1769076561) [ 1942.133064] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 1944.402126] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 05:09:25 (1769076565) [ 1945.245658] LNet: 93032:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1945.262752] LNetError: 93032:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1945.287463] LNet: Removed LNI 192.168.201.50@tcp [ 1946.229854] Key type .llcrypt unregistered [ 1946.233690] Key type ._llcrypt unregistered [ 1954.409297] Key type ._llcrypt registered [ 1954.413483] Key type .llcrypt registered [ 1954.725973] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1954.771225] alg: No test for adler32 (adler32-zlib) [ 1955.765555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1955.814849] LNet: Added LNI 192.168.201.50@tcp [8/256/0/180] [ 1963.485060] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.150@tcp [ 1968.972266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.201.150@tcp [ 1969.279569] LNet: 93445:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1969.287193] LNetError: 93445:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1969.312785] LNet: Removed LNI 192.168.201.50@tcp [ 1969.754400] Key type .llcrypt unregistered [ 1969.756218] Key type ._llcrypt unregistered [ 1981.894856] Key type ._llcrypt registered [ 1981.896961] Key type .llcrypt registered [ 1983.419569] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 1985.113789] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 1988.091476] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 05:10:09 (1769076609) [ 1988.730197] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1988.763524] alg: No test for adler32 (adler32-zlib) [ 1989.859693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1989.995203] LNet: 94530:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1990.763361] Key type .llcrypt unregistered [ 1990.765591] Key type ._llcrypt unregistered [ 1999.388945] Key type ._llcrypt registered [ 1999.397638] Key type .llcrypt registered [ 1999.799618] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1999.822630] alg: No test for adler32 (adler32-zlib) [ 2001.026841] LNet: Added LNI 192.168.201.50@tcp [8/256/0/180] [ 2001.034105] LNet: 94783:0:(config.c:1226:lnet_parse_routes()) Kernel parsing of LNet routes is deprecated. Consider converting to LNet YAML configuration. [ 2001.060069] LNet: 94783:0:(router.c:733:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 2001.164685] LNetError: 94795:0:(router.c:398:lnet_router_discovery_ping_reply()) Peer 192.168.201.51@tcp is being used as a gateway but routing feature is not turned on [ 2001.201808] LNetError: 94795:0:(lib-lnet.h:1337:lnet_set_route_aliveness()) route to o2ib through 192.168.201.51@tcp has gone from up to down [ 2001.725266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2001.743929] LNet: 94842:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2001.750133] LNetError: 94842:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2001.767805] LNet: Removed LNI 192.168.201.50@tcp [ 2001.793857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2001.900892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2001.923633] LNet: Added LNI 192.168.201.50@tcp [8/256/0/180] [ 2001.934785] LNet: 94849:0:(router.c:733:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 2001.949330] LNet: 94849:0:(router.c:733:lnet_add_route()) Skipped 4 previous similar messages [ 2001.963641] LNetError: 94849:0:(peer.c:368:lnet_peer_ni_del_locked()) Peer NI 192.168.201.51@tcp is a gateway. Can not delete it [ 2001.970470] LNetError: 94849:0:(peer.c:2217:lnet_del_peer_ni()) 192.168.201.51@tcp is a router. Can not be deleted [ 2001.996858] LNet: 94849:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2002.007238] LNetError: 94846:0:(router.c:398:lnet_router_discovery_ping_reply()) Peer 192.168.201.55@tcp is being used as a gateway but routing feature is not turned on [ 2002.024185] LNetError: 94846:0:(router.c:398:lnet_router_discovery_ping_reply()) Skipped 5 previous similar messages [ 2002.029902] LNetError: 94846:0:(lib-lnet.h:1337:lnet_set_route_aliveness()) route to o2ib through 192.168.201.55@tcp has gone from up to down [ 2002.047908] LNetError: 94846:0:(lib-lnet.h:1337:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 2011.478888] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 05:10:32 (1769076632) [ 2014.360466] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg150-server [ 2016.714409] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 05:10:38 (1769076638) [ 2019.254871] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg150-server [ 2021.168268] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 05:10:43 (1769076643) [ 2023.293265] Lustre: Lustre: Build Version: 2.17.50_3_gca5b8be [ 2025.512393] Key type lgssc registered [ 2027.061551] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2050.595732] Key type lgssc unregistered [ 2051.017569] LNet: 97272:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2051.036804] LNetError: 97272:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2051.064564] LNet: Removed LNI 192.168.201.50@tcp [ 2051.922374] Key type .llcrypt unregistered [ 2051.925127] Key type ._llcrypt unregistered [ 2068.851856] Key type ._llcrypt registered [ 2068.854179] Key type .llcrypt registered [ 2070.195674] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 05:11:32 (1769076692) [ 2070.561644] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2070.583815] alg: No test for adler32 (adler32-zlib) [ 2071.548611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2071.673716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2071.753788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2071.862624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2071.988072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2078.322325] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 05:11:40 (1769076700) [ 2078.654575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2078.681688] LNet: 98677:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2078.712572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2078.793631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2078.839080] LNet: Added LNI 192.168.201.50@tcp [8/256/0/180] [ 2080.071743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2080.134305] LNet: 98821:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2080.151376] LNetError: 98821:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2080.169661] LNet: Removed LNI 192.168.201.50@tcp [ 2080.231689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2080.339935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 2080.374513] LNet: Added LNI 192.168.201.50@tcp [8/256/0/180] [ 2080.793257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2080.813088] LNetError: 98850:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2080.832604] LNet: Removed LNI 192.168.201.50@tcp [ 2080.864337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2080.929324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 2080.957526] LNet: Added LNI 192.168.201.50@tcp [8/256/0/180] [ 2088.296493] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 05:11:50 (1769076710) [ 2088.560907] LNet: 99441:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2088.568137] LNet: 99441:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 2088.576941] LNetError: 99441:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2088.591231] LNet: Removed LNI 192.168.201.50@tcp [ 2089.607648] Key type .llcrypt unregistered [ 2089.609080] Key type ._llcrypt unregistered [ 2098.200537] Key type ._llcrypt registered [ 2098.206708] Key type .llcrypt registered [ 2098.591327] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2098.618191] alg: No test for adler32 (adler32-zlib) [ 2099.495296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2099.556702] LNet: Added LNI 192.168.201.50@tcp [8/256/0/180] [ 2099.561244] LNet: *** cfs_fail_loc=e003, val=100*** [ 2100.533883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.50@tcp [ 2100.567847] LNet: *** cfs_fail_loc=e003, val=100*** [ 2100.573869] LNet: Skipped 2 previous similar messages [ 2100.583124] Ping reply received with mapping: [ 2100.586660] found metadata reply NI[192.168.201.50@tcp] [ 2100.591586] 32 c9 a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 2100.595956] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 2100.600137] 5a 5a 5a 5a 5a 5a 5a 5a [ 2100.756339] LNet: 99835:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2100.763647] LNetError: 99835:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2100.782535] LNet: Removed LNI 192.168.201.50@tcp [ 2101.648415] Key type .llcrypt unregistered [ 2101.653539] Key type ._llcrypt unregistered [ 2114.727494] Key type ._llcrypt registered [ 2114.730727] Key type .llcrypt registered [ 2116.977321] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 05:12:17 (1769076737) [ 2117.706356] Key type .llcrypt unregistered [ 2117.708356] Key type ._llcrypt unregistered [ 2127.299136] Key type ._llcrypt registered [ 2127.303026] Key type .llcrypt registered [ 2127.634547] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2127.657722] alg: No test for adler32 (adler32-zlib) [ 2129.158638] LNetError: 100871:0:(api-ni.c:2732:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 2129.551582] Key type .llcrypt unregistered [ 2129.553818] Key type ._llcrypt unregistered [ 2136.438683] Key type ._llcrypt registered [ 2136.445696] Key type .llcrypt registered [ 2138.123754] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 05:12:39 (1769076759) [ 2138.697477] Key type .llcrypt unregistered [ 2138.701318] Key type ._llcrypt unregistered [ 2146.440792] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2146.448039] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2148.229272] Key type ._llcrypt registered [ 2148.234038] Key type .llcrypt registered [ 2148.481540] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2148.515349] alg: No test for adler32 (adler32-zlib) [ 2149.474502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2149.553365] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2160.567614] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 05:13:02 (1769076782) [ 2160.749864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2160.794285] LNet: 102637:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2160.805325] LNetError: 102637:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2160.847369] LNet: Removed LNI 10.1.2.3@tcp [ 2160.890451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2161.793961] Key type .llcrypt unregistered [ 2161.801041] Key type ._llcrypt unregistered [ 2170.048346] Key type ._llcrypt registered [ 2170.054046] Key type .llcrypt registered [ 2170.212381] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2170.224452] alg: No test for adler32 (adler32-zlib) [ 2171.282532] LNet: Added LNI 192.168.201.50@tcp [8/256/0/180] [ 2171.386645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.50@tcp [ 2171.567491] LNet: 102923:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2171.573122] LNetError: 102923:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2171.614994] LNet: Removed LNI 192.168.201.50@tcp [ 2172.265543] Key type .llcrypt unregistered [ 2172.272714] Key type ._llcrypt unregistered [ 2185.937988] Key type ._llcrypt registered [ 2185.939904] Key type .llcrypt registered [ 2187.752924] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 05:13:28 (1769076808) [ 2188.473361] Key type .llcrypt unregistered [ 2188.475691] Key type ._llcrypt unregistered [ 2196.236770] Key type ._llcrypt registered [ 2196.240554] Key type .llcrypt registered [ 2196.525836] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2196.557686] alg: No test for adler32 (adler32-zlib) [ 2205.739429] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 05:13:47 (1769076827) [ 2205.870782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2205.937188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2206.053378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2206.093320] LNet: Added LNI 192.168.201.50@tcp [8/256/0/180] [ 2206.726366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2206.754942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 2206.780146] LNet: Removed LNI 192.168.201.50@tcp [ 2207.100551] LNet: 104660:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2207.800404] Key type .llcrypt unregistered [ 2207.803730] Key type ._llcrypt unregistered [ 2223.805716] Key type ._llcrypt registered [ 2223.809973] Key type .llcrypt registered [ 2226.153661] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 05:14:06 (1769076846) [ 2226.594254] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2226.609070] alg: No test for adler32 (adler32-zlib) [ 2227.658964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2227.787180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2227.853683] LNet: Added LNI 192.168.201.50@tcp [8/256/0/180] [ 2229.239500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.50@tcp [ 2229.308702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.201.50@tcp [ 2229.327585] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 2229.331992] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 2229.334347] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2229, 0, 0 [ 2229.339641] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 2229.347732] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 2229.358242] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 2229.364977] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2229, 0, 0 [ 2229.377627] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 2229.383671] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2229.392664] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 2229.401866] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2229, 0, 2279 [ 2229.411363] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 2229.420465] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2229.429887] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 2229.436661] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2229, 0, 2279 [ 2229.448619] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 2239.600218] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 05:14:20 (1769076860) [ 2240.457357] LNet: 106198:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2240.468336] LNetError: 106198:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2240.490190] LNet: Removed LNI 192.168.201.50@tcp [ 2241.614674] Key type .llcrypt unregistered [ 2241.618581] Key type ._llcrypt unregistered [ 2249.863379] Key type ._llcrypt registered [ 2249.875116] Key type .llcrypt registered [ 2250.205810] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2250.220816] alg: No test for adler32 (adler32-zlib) [ 2251.183415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2251.218293] LNet: Added LNI 192.168.201.50@tcp [8/256/0/180] [ 2260.829324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.150@tcp [ 2271.688161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2271.747700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2271.796214] LNet: Added LNI 192.168.201.50@tcp1 [8/256/0/180] [ 2273.069125] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2273.314113] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2273.328979] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2273.343269] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2275.725490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 2275.755048] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 2287.683961] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 05:15:09 (1769076909) [ 2287.839432] LNet: 107693:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2287.845962] LNetError: 107693:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2287.869296] LNet: Removed LNI 192.168.201.50@tcp [ 2288.932725] LNet: Removed LNI 192.168.201.50@tcp1 [ 2288.934720] LNetError: 107693:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2288.946596] LNetError: 107693:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2288.964680] LNet: Removed LNI 10.1.2.3@tcp99 [ 2290.158884] Key type .llcrypt unregistered [ 2290.160592] Key type ._llcrypt unregistered [ 2297.103983] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2297.116583] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2298.622224] Key type ._llcrypt registered [ 2298.624545] Key type .llcrypt registered [ 2298.919289] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2298.934933] alg: No test for adler32 (adler32-zlib) [ 2299.877073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2299.979383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2300.035738] LNet: Added LNI 192.168.201.50@tcp [8/256/0/180] [ 2301.160135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2301.177146] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2304.185591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 2304.271618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2304.319333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 2304.333498] LNetError: 108434:0:(peer.c:2225:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 2304.360524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2304.402433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.50@tcp [ 2304.771814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2304.871354] LNet: 108454:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2304.886332] LNetError: 108454:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2304.896327] LNet: Removed LNI 192.168.201.50@tcp [ 2304.908287] LNet: Removed LNI 10.1.2.3@tcp [ 2305.877519] Key type .llcrypt unregistered [ 2305.883747] Key type ._llcrypt unregistered [ 2320.983830] Key type ._llcrypt registered [ 2320.988127] Key type .llcrypt registered [ 2322.805457] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 05:15:44 (1769076944) [ 2323.246528] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2323.263928] alg: No test for adler32 (adler32-zlib) [ 2324.326062] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2324.395614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2324.427393] LNet: Added LNI 192.168.201.50@tcp [8/256/0/180] [ 2325.297823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.50@tcp [ 2325.432566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg150-client@tcp [ 2333.020881] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 05:15:54 (1769076954) [ 2333.441239] LNet: 109984:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2333.449580] LNetError: 109984:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2333.481853] LNet: Removed LNI 192.168.201.50@tcp [ 2334.068506] Key type .llcrypt unregistered [ 2334.071542] Key type ._llcrypt unregistered [ 2341.691412] Key type ._llcrypt registered [ 2341.692840] Key type .llcrypt registered [ 2342.001924] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2342.033450] alg: No test for adler32 (adler32-zlib) [ 2343.102997] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 2343.172788] LNet: Added LNI 192.168.201.50@tcp [8/256/0/180] [ 2343.241738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 2351.143912] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 05:16:12 (1769076972) [ 2353.198650] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 2355.552631] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 05:16:16 (1769076976) [ 2357.855078] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 2359.838862] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 05:16:21 (1769076981) [ 2360.216448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2360.262172] LNet: 111400:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2360.275078] LNetError: 111400:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2360.289195] LNet: Removed LNI 192.168.201.50@tcp [ 2360.316522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2360.373220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2360.413761] LNet: Added LNI 192.168.201.50@tcp [8/256/0/180] [ 2361.264601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2361.325269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2369.030514] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 05:16:30 (1769076990) [ 2369.239687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2369.284537] LNet: 112109:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2369.303672] LNetError: 112109:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2369.326225] LNet: Removed LNI 192.168.201.50@tcp [ 2369.393981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2369.472855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 2369.531844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2370.590488] Key type .llcrypt unregistered [ 2370.592723] Key type ._llcrypt unregistered [ 2385.984810] Key type ._llcrypt registered [ 2385.989178] Key type .llcrypt registered [ 2388.079364] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 05:16:48 (1769077008) [ 2388.917675] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2388.951255] alg: No test for adler32 (adler32-zlib) [ 2390.075685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2390.132985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2390.153607] LNet: Added LNI 192.168.201.50@tcp [8/256/0/180] [ 2391.388439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2391.560477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.50@tcp [ 2400.484178] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 05:17:01 (1769077021) [ 2400.947188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2400.971938] LNet: 113663:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2400.979040] LNetError: 113663:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2400.997419] LNet: Removed LNI 192.168.201.50@tcp [ 2401.021512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2401.130522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 2401.175837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 2409.391937] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 05:17:10 (1769077030) [ 2409.533111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2409.595908] LNet: 114239:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2409.661212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2409.772401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2410.198308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2410.265513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2410.396369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 2411.057910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2411.135942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2411.298286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2411.357854] LNet: 114276:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2411.392974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2411.521739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2412.492912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2412.561957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2412.652525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 2413.103943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2413.149722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2421.614925] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 05:17:22 (1769077042) [ 2421.889183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2421.943439] LNet: 114877:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2422.004647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2422.098142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2422.133189] LNet: Added LNI 192.168.201.50@tcp [8/256/0/180] [ 2423.145165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2423.170096] LNet: Added LNI 192.168.201.50@tcp2 [8/256/0/180] [ 2424.476686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2424.646589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.50@tcp [ 2425.081770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2425.126305] LNetError: 115157:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2425.145929] LNet: Removed LNI 192.168.201.50@tcp [ 2425.166407] LNet: Removed LNI 192.168.201.50@tcp2 [ 2425.229219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2425.405188] LNet: Added LNI 192.168.201.50@tcp [8/256/0/180] [ 2425.431668] LNet: Added LNI 192.168.201.50@tcp2 [8/256/0/180] [ 2425.443899] LNet: 115163:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2426.449444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.50@tcp [ 2426.817860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2426.864786] LNet: 115201:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2426.870891] LNet: 115201:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 2426.887951] LNetError: 115201:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2426.910941] LNetError: 115201:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2426.937073] LNet: Removed LNI 192.168.201.50@tcp [ 2426.959337] LNet: Removed LNI 192.168.201.50@tcp2 [ 2427.073853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2427.232990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2427.285975] LNet: Added LNI 192.168.201.50@tcp [8/256/0/180] [ 2428.887964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2428.929921] LNet: Added LNI 192.168.201.50@tcp2 [8/256/0/180] [ 2431.105957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.50@tcp [ 2431.246907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2431.482880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.50@tcp [ 2432.075933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2432.150395] LNetError: 115485:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2432.159818] LNetError: 115485:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2432.174745] LNet: Removed LNI 192.168.201.50@tcp [ 2433.220451] LNet: Removed LNI 192.168.201.50@tcp2 [ 2433.291849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2433.458268] LNet: Added LNI 192.168.201.50@tcp [8/256/0/180] [ 2433.467854] LNet: Added LNI 192.168.201.50@tcp2 [8/256/0/180] [ 2433.473808] LNet: 115491:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2434.783646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.50@tcp [ 2435.277919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2435.346932] LNet: 115529:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2435.353412] LNet: 115529:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 2435.373903] LNet: Removed LNI 192.168.201.50@tcp [ 2436.389729] LNet: Removed LNI 192.168.201.50@tcp2 [ 2436.432426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2436.494084] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2436.512719] LNet: Added LNI 192.168.201.50@tcp [8/256/0/180] [ 2437.087387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2437.114940] LNet: Added LNI 192.168.201.50@tcp2 [8/256/0/180] [ 2438.106877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2438.164102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.50@tcp [ 2438.371776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2438.388658] LNetError: 115808:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2438.392542] LNetError: 115808:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 2438.397578] LNet: Removed LNI 192.168.201.50@tcp [ 2438.408752] LNet: Removed LNI 192.168.201.50@tcp2 [ 2438.439083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2438.517498] LNet: Added LNI 192.168.201.50@tcp [8/256/0/180] [ 2438.522358] LNet: Added LNI 192.168.201.50@tcp2 [8/256/0/180] [ 2438.527687] LNet: 115814:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2439.214769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.50@tcp [ 2439.570990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2440.673602] LNet: Removed LNI 192.168.201.50@tcp [ 2440.689611] LNet: Removed LNI 192.168.201.50@tcp2 [ 2440.709720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2440.755549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2440.800498] LNet: Added LNI 192.168.201.50@tcp [8/256/0/180] [ 2441.498374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2441.526939] LNet: Added LNI 192.168.201.50@tcp2 [8/256/0/180] [ 2442.565489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.50@tcp [ 2442.627529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2442.695614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.50@tcp [ 2442.941355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2443.999197] LNet: Removed LNI 192.168.201.50@tcp [ 2445.033948] LNet: Removed LNI 192.168.201.50@tcp2 [ 2445.079397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2445.173138] LNet: Added LNI 192.168.201.50@tcp [8/256/0/180] [ 2445.181818] LNet: Added LNI 192.168.201.50@tcp2 [8/256/0/180] [ 2445.190383] LNet: 116143:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2445.961462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.50@tcp [ 2446.140554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2446.155365] LNet: Removed LNI 192.168.201.50@tcp [ 2447.207163] LNet: Removed LNI 192.168.201.50@tcp2 [ 2447.265272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2447.393115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2447.437294] LNet: Added LNI 192.168.201.50@tcp [8/256/0/180] [ 2448.138909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2448.160684] LNet: Added LNI 192.168.201.50@tcp2 [8/256/0/180] [ 2449.158375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2449.210305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2449.286813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.50@tcp [ 2449.630565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2449.648324] LNetError: 116477:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2449.651946] LNetError: 116477:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 2450.719265] LNet: Removed LNI 192.168.201.50@tcp [ 2450.742628] LNet: Removed LNI 192.168.201.50@tcp2 [ 2450.820451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2450.923598] LNet: Added LNI 192.168.201.50@tcp [8/256/0/180] [ 2450.934928] LNet: Added LNI 192.168.201.50@tcp2 [8/256/0/180] [ 2450.940672] LNet: 116483:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2451.914673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.50@tcp [ 2452.389708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2452.434618] LNet: 116535:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2452.445549] LNet: 116535:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 5 previous similar messages [ 2453.471142] LNet: Removed LNI 192.168.201.50@tcp [ 2453.482507] LNet: Removed LNI 192.168.201.50@tcp2 [ 2453.499921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2453.606889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2453.645608] LNet: Added LNI 192.168.201.50@tcp [8/256/0/180] [ 2454.684563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2454.705678] LNet: Added LNI 192.168.201.50@tcp2 [8/256/0/180] [ 2455.776377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.50@tcp [ 2455.858089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2455.913741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2456.025155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.50@tcp [ 2456.522746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2457.567294] LNet: Removed LNI 192.168.201.50@tcp [ 2458.597637] LNet: Removed LNI 192.168.201.50@tcp2 [ 2458.656704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2458.795570] LNet: Added LNI 192.168.201.50@tcp [8/256/0/180] [ 2458.806785] LNet: Added LNI 192.168.201.50@tcp2 [8/256/0/180] [ 2459.770638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.50@tcp [ 2468.431349] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 2470.779700] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 05:18:12 (1769077092) [ 2471.156296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2471.193935] LNetError: 117604:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2471.199554] LNetError: 117604:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 2471.205607] LNet: Removed LNI 192.168.201.50@tcp [ 2472.230145] LNet: Removed LNI 192.168.201.50@tcp2 [ 2472.294659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2472.638826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2472.686531] LNet: Added LNI 192.168.201.50@tcp [8/256/0/180] [ 2473.657329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2473.684313] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2475.039248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2475.099103] LNet: Removed LNI 192.168.201.50@tcp [ 2475.115548] LNet: Removed LNI 10.1.2.3@tcp [ 2475.167662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2475.296567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 2475.384266] LNet: Added LNI 192.168.201.50@tcp [8/256/0/180] [ 2475.407238] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2476.367705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 2476.412599] LNet: Removed LNI 192.168.201.50@tcp [ 2476.432155] LNet: Removed LNI 10.1.2.3@tcp [ 2483.939954] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 05:18:25 (1769077105) [ 2484.085969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2484.186639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2484.340750] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2484.544523] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2484.559190] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2484.579780] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2487.028780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2487.053889] LNet: Added LNI 192.168.201.50@tcp [8/256/0/180] [ 2488.082953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2488.111579] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2489.949086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.50@tcp [ 2489.977251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 2490.027476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.201.50@tcp -m PUT -e local_timeout -r 1 [ 2490.075940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 2490.134760] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2490.138197] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2490.344780] LNetError: 118543:0:(lib-move.c:4947:lnet_parse()) 192.168.201.50@tcp, src 192.168.201.50@tcp: Dropping PUT (error -108 looking up sender) [ 2491.359765] LNet: 119002:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2491.373365] LNet: 119002:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 2492.449984] LNet: Removed LNI 192.168.201.50@tcp [ 2492.464912] LNet: Removed LNI 10.1.2.3@tcp [ 2493.078717] Key type .llcrypt unregistered [ 2493.081957] Key type ._llcrypt unregistered [ 2505.994815] Key type ._llcrypt registered [ 2505.997088] Key type .llcrypt registered [ 2507.403157] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 05:18:49 (1769077129) [ 2507.880977] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2507.899148] alg: No test for adler32 (adler32-zlib) [ 2508.940414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2509.375676] LNet: 119853:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2509.965382] Key type .llcrypt unregistered [ 2509.967210] Key type ._llcrypt unregistered [ 2517.731435] Key type ._llcrypt registered [ 2517.733384] Key type .llcrypt registered [ 2518.086497] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2518.098511] alg: No test for adler32 (adler32-zlib) [ 2519.108850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2519.155151] LNet: Added LNI 192.168.201.50@tcp [8/256/0/180] [ 2527.393350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.150@tcp [ 2527.652228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 2528.539796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.201.150@tcp [ 2528.904542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2528.944434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2529.027200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2529.078539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2529.127190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2529.171588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2529.209163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2529.234332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2529.249358] LNet: 120308:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2529.264580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2529.289360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2529.312467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2529.336794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2529.371082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2529.402232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2530.062909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2530.344282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2530.376915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2530.406697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2530.442284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2530.463119] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.150@tcp [ 2530.477088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2530.513727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2530.552511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2530.582770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2530.602773] LNet: 120425:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2530.619806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2530.652635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2530.682186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2530.724674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2530.787080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2530.843736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2531.541441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.201.150@tcp [ 2531.817804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2531.846722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2531.889854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2531.921991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2531.962694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2532.035716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2532.093555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2532.173705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2532.211958] LNet: 120543:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2532.244942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2532.297439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2532.352404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2532.404224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2532.446071] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2532.499248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2533.294978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2533.595227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2533.634444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2533.673814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2533.747929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2533.820516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2533.864310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2533.902792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2533.951645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2533.998831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2534.048373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2534.096897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2534.178964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2534.216823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2534.282344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2534.689742] LNet: 120705:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2534.698496] LNetError: 120705:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2534.722707] LNet: Removed LNI 192.168.201.50@tcp [ 2535.235921] Key type .llcrypt unregistered [ 2535.239746] Key type ._llcrypt unregistered [ 2555.930769] Key type ._llcrypt registered [ 2555.932918] Key type .llcrypt registered [ 2557.550687] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 05:19:38 (1769077178) [ 2558.061224] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2558.084657] alg: No test for adler32 (adler32-zlib) [ 2559.178753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2559.843789] LNet: 121591:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2560.861348] Key type .llcrypt unregistered [ 2560.867877] Key type ._llcrypt unregistered [ 2569.621608] Key type ._llcrypt registered [ 2569.625755] Key type .llcrypt registered [ 2569.874743] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2569.889161] alg: No test for adler32 (adler32-zlib) [ 2570.802733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2570.832436] LNet: Added LNI 192.168.201.50@tcp [8/256/0/180] [ 2579.856467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.150@tcp [ 2590.282536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2590.358691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2590.408452] LNet: Added LNI 192.168.201.50@tcp1 [8/256/0/180] [ 2591.670496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 2592.328873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.201.150@tcp [ 2592.358439] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.150@tcp [ 2592.636818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2592.687874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2592.737619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2592.791467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2592.856479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2592.977365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2593.030376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2593.100763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2593.130616] LNet: 122241:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2593.159978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2593.226720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2593.277283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2593.329611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2593.390776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2593.445376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2594.386968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2594.694133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2594.741342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2594.792738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2594.834880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2594.878984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2594.935777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2595.007517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2595.063143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2595.095551] LNet: 122359:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2595.125307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2595.183792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2595.263421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2595.321943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2595.369979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2595.414629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2596.223442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.201.150@tcp [ 2596.499271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2596.528148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2596.586949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2596.626798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2596.670959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2596.720727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2596.760853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2596.803913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2596.835808] LNet: 122476:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2596.852551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2596.904167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2596.975871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2597.019058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2597.070970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2597.118206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2598.084023] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2598.404293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2598.452932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2598.499710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2598.553534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2598.604465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2598.652705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2598.714325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2598.766230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2598.797666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2598.836379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2598.872733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2598.907583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2598.955756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2599.006505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2604.061984] LNet: 122674:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2604.071202] LNetError: 122674:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2604.078470] LNet: Removed LNI 192.168.201.50@tcp [ 2604.091816] LNet: Removed LNI 192.168.201.50@tcp1 [ 2604.496702] Key type .llcrypt unregistered [ 2604.499469] Key type ._llcrypt unregistered [ 2624.606614] Key type ._llcrypt registered [ 2624.610333] Key type .llcrypt registered [ 2626.110809] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 05:20:47 (1769077247) [ 2635.278529] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 05:20:56 (1769077256) [ 2643.656586] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 05:21:05 (1769077265) [ 2653.162575] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 05:21:14 (1769077274) [ 2662.717310] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 05:21:23 (1769077283) [ 2674.431079] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 05:21:35 (1769077295) [ 2682.654789] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 05:21:44 (1769077304) [ 2689.716384] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 05:21:51 (1769077311) [ 2698.610287] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 05:22:00 (1769077320) [ 2706.759417] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 05:22:08 (1769077328) [ 2713.739933] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 05:22:15 (1769077335) [ 2725.088341] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 05:22:26 (1769077346) [ 2735.125799] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 05:22:36 (1769077356) [ 2744.937978] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 05:22:46 (1769077366) [ 2753.500824] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 05:22:55 (1769077375) [ 2761.841701] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 05:23:03 (1769077383) [ 2770.429365] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 05:23:11 (1769077391) [ 2777.873882] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 05:23:19 (1769077399) [ 2778.562299] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2778.579802] alg: No test for adler32 (adler32-zlib) [ 2779.530773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2779.696227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2779.719523] LNet: Added LNI 192.168.201.50@tcp [8/256/0/180] [ 2781.383466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2781.424408] LNet: 133387:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2781.447919] LNetError: 133387:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2781.472489] LNet: Removed LNI 192.168.201.50@tcp [ 2781.498922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2781.575589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2781.610658] LNet: Added LNI 192.168.201.50@tcp [8/256/0/180] [ 2789.769646] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 05:23:31 (1769077411) [ 2798.985593] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 05:23:40 (1769077420) [ 2809.323242] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 05:23:50 (1769077430) [ 2817.806111] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 05:23:59 (1769077439) [ 2826.154598] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 05:24:07 (1769077447) [ 2838.835271] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 05:24:20 (1769077460) [ 2848.340681] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 05:24:29 (1769077469) [ 2858.929667] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 05:24:40 (1769077480) [ 2867.244807] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 05:24:49 (1769077489) [ 2874.776307] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 05:24:56 (1769077496) [ 2882.380214] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 05:25:04 (1769077504) [ 2890.196405] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 05:25:11 (1769077511) [ 2898.107462] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 2632 sec ========== 05:25:20 (1769077520) [ 2898.441259] LNet: 140855:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2898.448747] LNetError: 140855:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2898.467037] LNet: Removed LNI 192.168.201.50@tcp [ 2899.185907] Key type .llcrypt unregistered [ 2899.194691] Key type ._llcrypt unregistered [ 2914.101481] Key type ._llcrypt registered [ 2914.108971] Key type .llcrypt registered [ 2914.830753] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2914.848150] alg: No test for adler32 (adler32-zlib) [ 2915.983519] Lustre: Lustre: Build Version: 2.17.50_3_gca5b8be [ 2916.277695] LNet: Added LNI 192.168.201.50@tcp [8/256/0/180] [ 2917.935170] Key type lgssc registered [ 2919.406403] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2984.656440] Lustre: Mounted lustre-client [ 2991.152874] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3005.267210] LustreError: 143309:0:(lov_obd.c:783:lov_cleanup()) lustre-clilov-ffff9cc29332a800: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 3005.301176] LustreError: 143309:0:(obd_class.h:479:obd_check_dev()) Device 5 not setup [ 3005.353277] Lustre: Unmounted lustre-client [ 3043.051770] Key type lgssc unregistered [ 3043.410894] LNet: 143791:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3043.426779] LNetError: 143791:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3043.452906] LNet: Removed LNI 192.168.201.50@tcp [ 3044.574370] Key type .llcrypt unregistered [ 3044.578344] Key type ._llcrypt unregistered