[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 432919820 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.002399] x2apic enabled [ 0.003010] Switched APIC routing to physical x2apic. [ 0.004019] kvm-guest: setup PV IPIs [ 0.006917] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007029] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008018] pid_max: default: 32768 minimum: 301 [ 0.009163] LSM: Security Framework initializing [ 0.010063] Yama: becoming mindful. [ 0.011057] SELinux: Initializing. [ 0.012073] *** VALIDATE selinux *** [ 0.020830] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.025139] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.026162] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027134] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028157] *** VALIDATE tmpfs *** [ 0.030283] *** VALIDATE proc *** [ 0.031282] *** VALIDATE cgroup *** [ 0.032013] *** VALIDATE cgroup2 *** [ 0.034006] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036153] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037014] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038031] Spectre V2 : User space: Vulnerable [ 0.039009] Speculative Store Bypass: Vulnerable [ 0.042505] debug: unmapping init [mem 0xffffffff88259000-0xffffffff88260fff] [ 0.044217] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045744] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046025] ... version: 2 [ 0.047010] ... bit width: 48 [ 0.048011] ... generic registers: 4 [ 0.049015] ... value mask: 0000ffffffffffff [ 0.050011] ... max period: 00007fffffffffff [ 0.051014] ... fixed-purpose events: 3 [ 0.052016] ... event mask: 000000070000000f [ 0.053350] rcu: Hierarchical SRCU implementation. [ 0.055239] smp: Bringing up secondary CPUs ... [ 0.056641] x86: Booting SMP configuration: [ 0.057037] .... node #0, CPUs: #1 #2 #3 [ 0.060430] smp: Brought up 1 node, 4 CPUs [ 0.061930] smpboot: Max logical packages: 1 [ 0.062012] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.145558] node 0 deferred pages initialised in 81ms [ 0.150026] devtmpfs: initialized [ 0.152149] x86/mm: Memory block size: 128MB [ 0.154341] gcov: version magic: 0x41383552 [ 0.156141] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.157089] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.158299] pinctrl core: initialized pinctrl subsystem [ 0.159218] [ 0.159749] ************************************************************* [ 0.160021] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.161024] ** ** [ 0.162017] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.163018] ** ** [ 0.164016] ** This means that this kernel is built to expose internal ** [ 0.165012] ** IOMMU data structures, which may compromise security on ** [ 0.166016] ** your system. ** [ 0.167016] ** ** [ 0.168011] ** If you see this message and you are not debugging the ** [ 0.169007] ** kernel, report this immediately to your vendor! ** [ 0.170007] ** ** [ 0.171011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.172016] ************************************************************* [ 0.173890] NET: Registered protocol family 16 [ 0.174625] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.175082] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.176091] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.178045] cpuidle: using governor menu [ 0.180020] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.182412] PCI: Using configuration type 1 for base access [ 0.184097] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.194166] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.196032] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.199292] cryptd: max_cpu_qlen set to 1000 [ 0.202137] ACPI: Added _OSI(Module Device) [ 0.204015] ACPI: Added _OSI(Processor Device) [ 0.205016] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.207016] ACPI: Added _OSI(Processor Aggregator Device) [ 0.212447] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.218358] ACPI: Interpreter enabled [ 0.219047] ACPI: PM: (supports S0 S3 S4 S5) [ 0.219926] ACPI: Using IOAPIC for interrupt routing [ 0.221092] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.223347] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.231970] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.233042] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.235022] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.238082] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.242439] acpiphp: Slot [2] registered [ 0.243106] acpiphp: Slot [5] registered [ 0.245148] acpiphp: Slot [6] registered [ 0.246154] acpiphp: Slot [3] registered [ 0.248126] acpiphp: Slot [4] registered [ 0.250182] acpiphp: Slot [7] registered [ 0.251136] acpiphp: Slot [8] registered [ 0.253127] acpiphp: Slot [9] registered [ 0.255122] acpiphp: Slot [10] registered [ 0.256162] acpiphp: Slot [11] registered [ 0.258142] acpiphp: Slot [12] registered [ 0.260128] acpiphp: Slot [13] registered [ 0.262145] acpiphp: Slot [14] registered [ 0.263139] acpiphp: Slot [15] registered [ 0.265187] acpiphp: Slot [16] registered [ 0.267130] acpiphp: Slot [17] registered [ 0.268120] acpiphp: Slot [18] registered [ 0.270153] acpiphp: Slot [19] registered [ 0.271162] acpiphp: Slot [20] registered [ 0.273144] acpiphp: Slot [21] registered [ 0.274126] acpiphp: Slot [22] registered [ 0.275151] acpiphp: Slot [23] registered [ 0.277118] acpiphp: Slot [24] registered [ 0.279164] acpiphp: Slot [25] registered [ 0.280121] acpiphp: Slot [26] registered [ 0.282143] acpiphp: Slot [27] registered [ 0.283120] acpiphp: Slot [28] registered [ 0.284170] acpiphp: Slot [29] registered [ 0.285242] acpiphp: Slot [30] registered [ 0.287245] acpiphp: Slot [31] registered [ 0.289090] PCI host bridge to bus 0000:00 [ 0.291027] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.293030] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.295035] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.298026] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.301061] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.304042] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.305286] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.309211] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.312617] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.320015] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.325101] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.327025] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.330033] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.332023] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.335400] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.342117] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.348185] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.351745] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.358021] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.370019] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.375018] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.381577] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.388022] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.396023] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.414020] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.424714] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.448031] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.454024] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.470017] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.480636] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.482339] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.483280] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.485289] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.487248] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.490112] iommu: Default domain type: Passthrough [ 0.493297] SCSI subsystem initialized [ 0.494131] ACPI: bus type USB registered [ 0.495155] usbcore: registered new interface driver usbfs [ 0.497105] usbcore: registered new interface driver hub [ 0.499125] usbcore: registered new device driver usb [ 0.501200] pps_core: LinuxPPS API ver. 1 registered [ 0.502012] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.505141] PTP clock support registered [ 0.508011] EDAC MC: Ver: 3.0.0 [ 0.510216] PCI: Using ACPI for IRQ routing [ 0.511955] NetLabel: Initializing [ 0.512021] NetLabel: domain hash size = 128 [ 0.513013] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.514274] NetLabel: unlabeled traffic allowed by default [ 0.516399] vgaarb: loaded [ 0.518330] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.520025] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.525564] clocksource: Switched to clocksource kvm-clock [ 0.631625] VFS: Disk quotas dquot_6.6.0 [ 0.633211] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.634734] *** VALIDATE ramfs *** [ 0.635572] *** VALIDATE hugetlbfs *** [ 0.636625] pnp: PnP ACPI init [ 0.639197] pnp: PnP ACPI: found 6 devices [ 0.660509] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.664084] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.666457] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.668812] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.671404] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.673873] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.676574] NET: Registered protocol family 2 [ 0.679128] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.684217] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.687850] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.693332] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.696701] TCP: Hash tables configured (established 65536 bind 65536) [ 0.699680] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.702685] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.705713] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.708814] NET: Registered protocol family 1 [ 0.711366] RPC: Registered named UNIX socket transport module. [ 0.713206] RPC: Registered udp transport module. [ 0.714790] RPC: Registered tcp transport module. [ 0.716668] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.719114] NET: Registered protocol family 44 [ 0.720360] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.721859] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.723886] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.725934] PCI: CLS 0 bytes, default 64 [ 0.727572] Unpacking initramfs... [ 2.073487] debug: unmapping init [mem 0xffff97ddbcc64000-0xffff97ddbffcffff] [ 2.077166] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.079184] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.081471] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.555437] Initialise system trusted keyrings [ 2.557236] Key type blacklist registered [ 2.559134] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.568409] zbud: loaded [ 2.571424] *** VALIDATE nfs *** [ 2.572653] *** VALIDATE nfs4 *** [ 2.574143] pstore: using deflate compression [ 2.577128] Platform Keyring initialized [ 2.679542] NET: Registered protocol family 38 [ 2.681464] Key type asymmetric registered [ 2.682704] Asymmetric key parser 'x509' registered [ 2.684474] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.687578] io scheduler mq-deadline registered [ 2.689505] io scheduler kyber registered [ 2.691032] io scheduler bfq registered [ 2.693073] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.696426] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.698836] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.701331] ACPI: Power Button [PWRF] [ 2.706563] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.712810] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.732956] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.760655] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.787832] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.794609] Non-volatile memory driver v1.3 [ 2.796134] Linux agpgart interface v0.103 [ 2.829618] virtio_blk virtio1: [vda] 145904 512-byte logical blocks (74.7 MB/71.2 MiB) [ 2.832586] vda: detected capacity change from 0 to 74702848 [ 2.848933] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 2.851697] vdb: detected capacity change from 0 to 1073741824 [ 2.857615] libphy: Fixed MDIO Bus: probed [ 2.864927] usbcore: registered new interface driver usbserial_generic [ 2.867396] usbserial: USB Serial support registered for generic [ 2.869738] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 2.874107] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 2.875495] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 2.877594] mousedev: PS/2 mouse device common for all mice [ 2.880316] rtc_cmos 00:05: RTC can wake from S4 [ 2.882621] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 2.886813] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 2.890386] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 2.893073] rtc_cmos 00:05: registered as rtc0 [ 2.894751] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 2.897299] intel_pstate: CPU model not supported [ 2.899934] hid: raw HID events driver (C) Jiri Kosina [ 2.901638] usbcore: registered new interface driver usbhid [ 2.903638] usbhid: USB HID core driver [ 2.904802] drop_monitor: Initializing network drop monitor service [ 2.906798] Initializing XFRM netlink socket [ 2.908742] NET: Registered protocol family 10 [ 2.912422] Segment Routing with IPv6 [ 2.913415] NET: Registered protocol family 17 [ 2.915105] mpls_gso: MPLS GSO support [ 2.921212] RAS: Correctable Errors collector initialized. [ 2.923220] AVX version of gcm_enc/dec engaged. [ 2.924941] AES CTR mode by8 optimization enabled [ 2.987784] sched_clock: Marking stable (2987707814, 0)->(3876791916, -889084102) [ 2.990283] registered taskstats version 1 [ 2.992829] Loading compiled-in X.509 certificates [ 2.994322] zswap: loaded using pool lzo/zbud [ 3.013832] Key type big_key registered [ 3.023119] Key type encrypted registered [ 3.024155] ima: No TPM chip found, activating TPM-bypass! [ 3.025446] ima: Allocated hash algorithm: sha1 [ 3.026597] ima: No architecture policies found [ 3.027801] evm: Initialising EVM extended attributes: [ 3.029205] evm: security.selinux [ 3.030048] evm: security.ima [ 3.030699] evm: security.capability [ 3.031544] evm: HMAC attrs: 0x1 [ 3.033207] rtc_cmos 00:05: setting system clock to 2026-08-15 21:04:31 UTC (1786827871) [ 3.037856] debug: unmapping init [mem 0xffffffff89203000-0xffffffff893fffff] [ 3.040076] debug: unmapping init [mem 0xffffffff87f82000-0xffffffff88258fff] [ 3.048097] Write protecting the kernel read-only data: 28672k [ 3.050717] debug: unmapping init [mem 0xffffffff86603000-0xffffffff867fffff] [ 3.052831] debug: unmapping init [mem 0xffffffff86f14000-0xffffffff86ffffff] [ 3.082130] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.089486] systemd[1]: Detected virtualization kvm. [ 3.090633] systemd[1]: Detected architecture x86-64. [ 3.091800] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.117305] systemd[1]: No hostname configured. [ 3.118425] systemd[1]: Set hostname to . [ 3.120061] random: systemd: uninitialized urandom read (16 bytes read) [ 3.121644] systemd[1]: Initializing machine ID from random generator. [ 3.156411] random: ln: uninitialized urandom read (6 bytes read) [ 3.237547] random: systemd: uninitialized urandom read (16 bytes read) [ 3.239428] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 3.242629] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 3.245513] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... Starting Journal Service... Starting Apply Kernel Variables... Starting Setup Virtual Console... [ OK ] Reached target Slices. [ OK ] Reached target Local File Systems. [ OK ] Started Memstrack Anylazing Service. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Swap. Starting Create Volatile Files and Directories... [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 3.810798] device-mapper: uevent: version 1.0.3 [ 3.812536] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 4.415534] virtio_net virtio0 ens2: renamed from eth0 [ 4.460908] scsi host0: ata_piix [ 4.468806] scsi host1: ata_piix [ 4.470343] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 4.473841] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 8.759074] dracut-initqueue[586]: RTNETLINK answers: File exists [ 9.536740] random: crng init done [ 9.538246] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 9.773225] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Timers. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. Stopping udev Kernel Device Manager... [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 10.756922] printk: systemd: 26 output lines suppressed due to ratelimiting [ 10.980588] SELinux: Disabled at runtime. [ 11.039105] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.044345] systemd[1]: Detected virtualization kvm. [ 11.045603] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 11.499220] systemd[1]: initrd-switch-root.service: Succeeded. [ 11.502808] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 11.507712] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 11.511206] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 11.513645] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 11.519982] systemd[1]: Starting Journal Service... Starting Journal Service... [ 11.524019] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Switch Root. [ OK ] Started Forward Password Requests to Wall Directory Watch. Starting Apply Kernel Variables... Starting Remount Root and Kernel File Systems... [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice system-getty.slice. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on Process Core Dump Socket. Activating swap /dev/disk/by-label/SWAP... Mounting Kernel Debug File System... [ OK ] Reached target Paths. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ 11.634182] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Stopped target Initrd Root File System. Mounting POSIX Message Queue File System... [ OK ] Stopped target Initrd File Systems. Mounting Huge Pages File System... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Started Journal Service. [ OK ] Started Apply Kernel Variables. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Huge Pages File System. [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [[ 11.903087] squashfs: version 4.0 (2009/01/31) Phillip Lougher  OK ] Mounted /mnt. [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 12.200055] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 12.241450] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 12.312268] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 12.327621] EDAC sbridge: Ver: 1.1.2 [ 13.353752] Key type dns_resolver registered [ 13.647650] NFS: Registering the id_resolver key type [ 13.649354] Key type id_resolver registered [ 13.650685] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Login Service... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg225-client login: [ 42.719631] libcfs: loading out-of-tree module taints kernel. [ 42.808443] Key type ._llcrypt registered [ 42.811631] Key type .llcrypt registered [ 43.016611] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 43.024397] alg: No test for adler32 (adler32-zlib) [ 44.036745] Lustre: Lustre: Build Version: 2.17.57_3_g928f38d [ 44.348690] LNet: Added LNI 192.168.202.25@tcp [8/256/0/180] [ 45.968234] Key type lgssc registered [ 46.617175] Lustre: Echo OBD driver; http://www.lustre.org/ [ 166.768220] Lustre: Mounted lustre-client - version 2.17.57_3_g928f38d [ 171.874690] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 181.719130] hrtimer: interrupt took 4119672 ns [ 187.213747] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing check_logdir /tmp/testlogs/ [ 192.484440] Lustre: lustre-OST0000-osc-ffff97de08f92800: disconnect after 23s idle [ 193.792597] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing yml_node [ 199.441266] Lustre: DEBUG MARKER: Client: 2.17.57.3 [ 202.427409] Lustre: DEBUG MARKER: MDS: 2.17.57.3 [ 205.594541] Lustre: DEBUG MARKER: OSS: 2.17.57.3 [ 207.507415] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Sat Aug 15 17:07:54 EDT 2026 [ 225.263303] Lustre: Unmounted lustre-client [ 245.422606] Key type lgssc unregistered [ 245.722369] LNet: 8406:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 245.734590] LNetError: 8406:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 245.767384] LNet: Removed LNI 192.168.202.25@tcp [ 246.450248] Key type .llcrypt unregistered [ 246.451408] Key type ._llcrypt unregistered [ 272.572264] Key type ._llcrypt registered [ 272.579748] Key type .llcrypt registered [ 272.918677] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 272.948260] alg: No test for adler32 (adler32-zlib) [ 274.025927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 274.103917] LNet: Added LNI 192.168.202.25@tcp [8/256/0/180] [ 274.134703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 278.297318] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 278.480628] LNet: 9292:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 278.485366] LNetError: 9292:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 278.498357] LNet: Removed LNI 192.168.202.25@tcp [ 279.153264] Key type .llcrypt unregistered [ 279.154937] Key type ._llcrypt unregistered [ 285.931900] Key type ._llcrypt registered [ 285.933617] Key type .llcrypt registered [ 287.321607] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 17:09:14 (1786828154) [ 287.862310] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 287.875519] alg: No test for adler32 (adler32-zlib) [ 288.847545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 288.983955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 289.005655] LNet: 9712:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 289.036310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 289.133597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 289.170650] LNet: 9719:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 295.635673] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 17:09:22 (1786828162) [ 295.836814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 295.868883] LNet: 10296:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 295.893894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 295.980450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 302.286923] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 17:09:29 (1786828169) [ 302.517457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 302.561480] LNet: 10883:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 302.603141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 302.739560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 309.755529] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 17:09:36 (1786828176) [ 309.946376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 309.981740] LNet: 11467:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 310.006082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 310.080873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 316.612434] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 17:09:43 (1786828183) [ 316.802485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 316.840235] LNet: 12050:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 316.870313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 316.939994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 317.290706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 323.544499] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 17:09:50 (1786828190) [ 323.914878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 323.955247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 324.032420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 324.453771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 325.017419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 332.138291] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 17:09:59 (1786828199) [ 332.339816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 332.402774] LNet: 13239:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 332.414501] LNet: 13239:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 332.434961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 332.515148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 340.344222] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 17:10:07 (1786828207) [ 340.567736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 340.607670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 340.712850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 340.754181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 341.375322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 341.428758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 341.787773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 341.823644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 342.348369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 342.406653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 342.906777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 342.977357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 343.563286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 343.608367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 344.257229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 344.287507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 344.690939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 344.733773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 345.208092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 345.251615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 345.718900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 345.765804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 346.360750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 346.402409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 346.877934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 346.925447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 347.383928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 347.450074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 354.860488] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 17:10:21 (1786828221) [ 355.040683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 355.078720] LNet: 14508:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 355.086545] LNet: 14508:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 355.109970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 355.162429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 355.194189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 361.818728] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 17:10:28 (1786828228) [ 362.049403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 362.111307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 362.207322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 362.245453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 368.140210] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 17:10:35 (1786828235) [ 368.248977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 368.274691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 368.357939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 368.391892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 373.925973] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 17:10:41 (1786828241) [ 374.228322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 374.253884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 374.308660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 374.334607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 381.189505] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 17:10:48 (1786828248) [ 381.269825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 381.338653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 381.516862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 381.557248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 389.033576] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 17:10:56 (1786828256) [ 389.076935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 389.093267] LNet: 17433:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 389.097766] LNet: 17433:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 389.124561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 389.269951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 389.302099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 398.848790] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 17:11:05 (1786828265) [ 399.498920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 399.549481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 399.609033] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 400.076907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 407.229590] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 17:11:14 (1786828274) [ 407.683385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 407.723686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 407.954165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 408.210430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 414.005034] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 17:11:21 (1786828281) [ 414.097647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 414.190671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 414.408864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 414.696606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 421.948905] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 17:11:29 (1786828289) [ 422.053738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 422.102779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 422.692116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 422.873575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 429.648837] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 17:11:37 (1786828297) [ 429.914396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 429.953264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 430.034417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 430.075941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 436.149636] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 17:11:43 (1786828303) [ 436.243637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 436.295048] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 436.379909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 436.422473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 443.528128] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 17:11:50 (1786828310) [ 443.588151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 443.637176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 443.765257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 451.105130] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 17:11:57 (1786828317) [ 451.209910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 451.283901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 451.372269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 458.619512] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 17:12:05 (1786828325) [ 458.731735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 458.779441] LNet: 22872:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 458.784878] LNet: 22872:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 8 previous similar messages [ 458.813418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 458.920903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 459.272779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 465.492888] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 17:12:12 (1786828332) [ 465.625801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 465.687195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 465.816221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 465.846893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 473.683700] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 17:12:20 (1786828340) [ 473.796635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 473.857109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 474.009443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 474.041809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 482.829635] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 17:12:29 (1786828349) [ 482.958849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 483.053430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 483.214280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 483.249971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 490.043839] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 17:12:37 (1786828357) [ 490.372668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 490.418764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 490.494666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 490.535681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 497.900845] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 17:12:44 (1786828364) [ 498.001076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 498.063691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 498.170830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 498.266384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 498.298157] LNetError: 25829:0:(peer.c:2362:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 498.354728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 498.408720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 505.666661] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 17:12:52 (1786828372) [ 505.753992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 505.804762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 505.897794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.7zfE0xZ9S6 [ 505.993928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.7zfE0xZ9S6 [ 506.036853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.7zfE0xZ9S6 [ 512.447473] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 17:12:59 (1786828379) [ 512.657643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 512.688209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 512.718614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 512.754956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 512.806214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 518.657364] Lustre: DEBUG MARKER: == sanity-lnet test 40: LNetAddPeer merges a pre-existing non-primary peer NID ========================================================== 17:13:05 (1786828385) [ 518.866754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 518.908795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 518.981193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 519.003051] LNet: Added LNI 192.168.202.25@tcp [8/256/0/180] [ 520.426770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.25@tcp [ 521.647487] Lustre: Lustre: Build Version: 2.17.57_3_g928f38d [ 523.560323] Key type lgssc registered [ 524.746328] Lustre: Echo OBD driver; http://www.lustre.org/ [ 528.447323] LNetError: 27591:0:(lib-move.c:2158:lnet_handle_find_routed_path()) peer 192.168.202.25@tcp99 has no available nets [ 531.061506] Key type lgssc unregistered [ 531.439798] LNetError: 28702:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 531.458702] LNet: Removed LNI 192.168.202.25@tcp [ 532.065632] Key type .llcrypt unregistered [ 532.069269] Key type ._llcrypt unregistered [ 539.188681] Key type ._llcrypt registered [ 539.190810] Key type .llcrypt registered [ 540.668874] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 17:13:27 (1786828407) [ 540.981415] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 540.998635] alg: No test for adler32 (adler32-zlib) [ 541.926656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 542.016846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 544.778162] LNet: Message forwarding will be enabled in 2s [ 547.874585] LNet: Message forwarding enabled [ 548.330820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 548.355196] LNet: Message forwarding will stop in 2s to 6s [ 552.856328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 553.353717] LNet: Message forwarding disabled [ 555.011687] LNet: Message forwarding will be enabled in 2s [ 558.496098] LNet: Message forwarding enabled [ 561.674568] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 17:13:48 (1786828428) [ 561.751595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 561.788566] LNet: 30102:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 562.844552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 562.907348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 568.032996] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 17:13:55 (1786828435) [ 568.184529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 568.232144] LNet: 30692:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 568.253807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 568.362596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 568.404930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 568.444676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 568.485612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 568.520738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 568.567176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 568.618300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 568.649988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 568.698263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 568.730494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 568.779512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 568.810577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 568.851483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 568.883219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 568.928371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 569.004156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 569.050735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 569.073697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 569.104898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 569.127573] LNetError: 30736:0:(router.c:625:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 575.016226] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 17:14:02 (1786828442) [ 575.213079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 575.233665] LNet: 31307:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 575.253347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 575.335407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 581.427834] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 17:14:08 (1786828448) [ 583.264802] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg225-server [ 584.869826] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 17:14:12 (1786828452) [ 584.963779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 584.989929] LNet: 32231:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 585.029537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 585.120515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 585.157949] LNet: Added LNI 192.168.202.25@tcp [8/256/0/180] [ 585.958306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.202.[26-30]@tcp [ 585.998714] LNet: 32370:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 586.077566] LNetError: 32240:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.202.28@tcp is being used as a gateway but routing feature is not turned on [ 586.092767] LNetError: 32240:0:(lib-lnet.h:1279:lnet_set_route_aliveness()) route to tcp1 through 192.168.202.28@tcp has gone from up to down [ 587.063205] LNetError: 32240:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.202.30@tcp is being used as a gateway but routing feature is not turned on [ 587.082375] LNetError: 32240:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 588.075392] LNetError: 32240:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.202.26@tcp is being used as a gateway but routing feature is not turned on [ 588.109854] LNetError: 32240:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 4 previous similar messages [ 590.130886] LNetError: 32240:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.202.26@tcp is being used as a gateway but routing feature is not turned on [ 590.151969] LNetError: 32240:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 592.833596] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 17:14:19 (1786828459) [ 594.212840] LNetError: 32240:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.202.29@tcp is being used as a gateway but routing feature is not turned on [ 594.224733] LNetError: 32240:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 19 previous similar messages [ 594.806871] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg225-server [ 596.522786] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 17:14:23 (1786828463) [ 596.783432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 596.801303] LNet: 33306:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 596.808651] LNetError: 33306:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 596.818377] LNet: Removed LNI 192.168.202.25@tcp [ 596.857542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 596.938694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 596.966549] LNet: Added LNI 192.168.202.25@tcp [8/256/0/180] [ 597.853922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.202.[26-31/2]@tcp [ 597.881818] LNet: 33423:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 597.894873] LNet: 33423:0:(router.c:729:lnet_add_route()) Skipped 4 previous similar messages [ 597.920946] LNetError: 33310:0:(lib-lnet.h:1279:lnet_set_route_aliveness()) route to tcp103 through 192.168.202.26@tcp has gone from up to down [ 597.929195] LNetError: 33310:0:(lib-lnet.h:1279:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 597.962165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.202.[26-31/2]@tcp [ 604.475584] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 17:14:31 (1786828471) [ 604.736049] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 604.757193] LNetError: 34015:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 604.773368] LNet: Removed LNI 192.168.202.25@tcp [ 604.828698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 604.950668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 605.008204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 605.032474] LNet: 34029:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 605.036927] LNet: 34029:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 605.076380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 605.145648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 605.215567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 605.251954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 605.306844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 605.387923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 605.448329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 605.519352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 605.607378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 605.642174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 605.714879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 605.785148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 605.813615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 605.851731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 605.911371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 605.966191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 606.041115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 606.163100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 606.195767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 606.258643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 606.353429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 606.412441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 606.476873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 606.544582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 606.585740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 606.668372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 606.752262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 606.781692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 606.828293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 606.880175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 606.910308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 606.957984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 613.434813] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 17:14:40 (1786828480) [ 615.419754] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg225-server [ 617.275072] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 17:14:44 (1786828484) [ 619.303820] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg225-server [ 621.273123] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 17:14:48 (1786828488) [ 621.359458] LNet: 35408:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 621.369559] LNet: 35408:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 10 previous similar messages [ 622.101313] Key type .llcrypt unregistered [ 622.103408] Key type ._llcrypt unregistered [ 628.507620] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 628.760760] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 628.763935] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 628.776272] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 631.363831] Key type ._llcrypt registered [ 631.366107] Key type .llcrypt registered [ 631.597700] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 631.612135] alg: No test for adler32 (adler32-zlib) [ 632.534690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 632.596434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 632.613118] LNet: Added LNI 192.168.202.25@tcp [8/256/0/180] [ 633.211909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 633.233303] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 634.880694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 634.893227] LNet: Removed LNI 10.1.2.3@tcp [ 634.984510] LNet: 36143:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 634.992791] LNetError: 36143:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 635.006861] LNet: Removed LNI 192.168.202.25@tcp [ 635.548452] Key type .llcrypt unregistered [ 635.550078] Key type ._llcrypt unregistered [ 647.491440] Key type ._llcrypt registered [ 647.494255] Key type .llcrypt registered [ 648.885978] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 17:15:15 (1786828515) [ 649.739508] Key type .llcrypt unregistered [ 649.741432] Key type ._llcrypt unregistered [ 656.954374] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 657.144120] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 657.150668] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 657.159680] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 659.800049] Key type ._llcrypt registered [ 659.801510] Key type .llcrypt registered [ 660.023639] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 660.033762] alg: No test for adler32 (adler32-zlib) [ 660.925765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 660.970510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 660.991355] LNet: Added LNI 192.168.202.25@tcp [8/256/0/180] [ 661.613825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 661.638464] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 664.392507] LNet: 37643:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 664.400827] LNetError: 37643:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 664.408832] LNet: Removed LNI 10.1.2.3@tcp [ 664.419512] LNet: Removed LNI 192.168.202.25@tcp [ 664.922567] Key type .llcrypt unregistered [ 664.924337] Key type ._llcrypt unregistered [ 677.793682] Key type ._llcrypt registered [ 677.796159] Key type .llcrypt registered [ 679.953195] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 17:15:46 (1786828546) [ 680.849301] Key type .llcrypt unregistered [ 680.850823] Key type ._llcrypt unregistered [ 690.413079] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 690.730147] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 690.738978] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 690.745212] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 692.559199] Key type ._llcrypt registered [ 692.566904] Key type .llcrypt registered [ 692.958995] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 692.992537] alg: No test for adler32 (adler32-zlib) [ 694.218669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 694.321802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 694.365196] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 697.187915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 697.204646] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 699.761736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 699.782558] LNet: Removed LNI 10.1.2.3@tcp [ 699.816762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 699.841416] LNet: Removed LNI 10.1.2.31@tcp [ 699.862718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 699.884818] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 700.978547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 701.002353] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 702.211584] LNet: 39559:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 702.219375] LNetError: 39559:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg:0 not found [ 702.226565] LNet: Removed LNI 10.1.2.31@tcp [ 702.238352] LNet: Removed LNI 10.1.2.3@tcp [ 703.139762] Key type .llcrypt unregistered [ 703.144707] Key type ._llcrypt unregistered [ 716.346609] Key type ._llcrypt registered [ 716.348101] Key type .llcrypt registered [ 717.775442] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 17:16:24 (1786828584) [ 718.226209] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 718.257681] alg: No test for adler32 (adler32-zlib) [ 719.455603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 719.512965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 719.538706] LNet: Added LNI 192.168.202.25@tcp [8/256/0/180] [ 720.420948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 720.459643] LNet: 40516:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 720.473287] LNetError: 40516:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 720.513936] LNet: Removed LNI 192.168.202.25@tcp [ 720.549785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 720.654064] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.202.25@tcp [ 720.705835] LNet: Added LNI 192.168.202.25@tcp [8/256/0/180] [ 721.554049] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.202.25@tcp [ 721.576618] LNet: Removed LNI 192.168.202.25@tcp [ 721.653134] LNet: 40651:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 722.549513] Key type .llcrypt unregistered [ 722.551274] Key type ._llcrypt unregistered [ 735.091301] Key type ._llcrypt registered [ 735.095885] Key type .llcrypt registered [ 736.383598] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 17:16:43 (1786828603) [ 738.603815] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg225-server [ 740.348224] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 17:16:47 (1786828607) [ 741.077344] Key type .llcrypt unregistered [ 741.084111] Key type ._llcrypt unregistered [ 748.016100] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 748.316578] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 748.324107] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 748.335410] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 750.984957] Key type ._llcrypt registered [ 750.990609] Key type .llcrypt registered [ 751.183202] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 751.205964] alg: No test for adler32 (adler32-zlib) [ 752.098787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 752.166484] LNet: Added LNI 192.168.202.25@tcp1 [8/256/0/180] [ 752.756593] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 754.485059] LNet: 42496:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 754.491837] LNetError: 42496:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 754.498828] LNet: Removed LNI 192.168.202.25@tcp1 [ 754.508193] LNet: Removed LNI 10.1.2.3@tcp2 [ 755.215551] Key type .llcrypt unregistered [ 755.219394] Key type ._llcrypt unregistered [ 768.248813] Key type ._llcrypt registered [ 768.250238] Key type .llcrypt registered [ 769.394203] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 17:17:16 (1786828636) [ 769.995218] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 770.016155] alg: No test for adler32 (adler32-zlib) [ 770.930614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 770.985279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 771.006582] LNet: Added LNI 192.168.202.25@tcp [8/256/0/180] [ 772.087702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 772.115555] LNet: 43471:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 772.120885] LNetError: 43471:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 772.132657] LNet: Removed LNI 192.168.202.25@tcp [ 772.150194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 772.262359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 772.298589] LNet: Added LNI 192.168.202.25@tcp [8/256/0/180] [ 779.126641] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 17:17:26 (1786828646) [ 779.702242] LNet: 44062:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 779.710971] LNetError: 44062:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 779.750760] LNet: Removed LNI 192.168.202.25@tcp [ 780.425582] Key type .llcrypt unregistered [ 780.428795] Key type ._llcrypt unregistered [ 787.816895] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 788.113849] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 788.123185] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 788.133665] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 790.674848] Key type ._llcrypt registered [ 790.676893] Key type .llcrypt registered [ 790.841649] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 790.853260] alg: No test for adler32 (adler32-zlib) [ 791.775781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 791.843669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 791.871243] LNet: Added LNI 192.168.202.25@tcp [8/256/0/180] [ 792.809232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 792.830143] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 795.151175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 795.164970] LNet: 44810:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 795.169608] LNetError: 44810:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 795.173994] LNet: Removed LNI 10.1.2.3@tcp [ 795.183517] LNet: Removed LNI 192.168.202.25@tcp [ 795.204459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 795.262677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 795.280295] LNet: Added LNI 192.168.202.25@tcp [8/256/0/180] [ 795.282471] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 795.689657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 795.718291] LNet: 44837:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 795.730983] LNetError: 44837:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 795.746190] LNetError: 44837:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 795.752509] LNet: Removed LNI 10.1.2.3@tcp [ 795.769599] LNet: Removed LNI 192.168.202.25@tcp [ 795.786228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 795.840455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 795.881735] LNet: Added LNI 192.168.202.25@tcp [8/256/0/180] [ 795.888508] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 796.362210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 796.384432] LNet: Added LNI 192.168.202.25@tcp2 [8/256/0/180] [ 797.334341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 797.354371] LNet: 44999:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 797.360341] LNetError: 44999:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 797.365280] LNetError: 44999:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 797.369796] LNet: Removed LNI 10.1.2.3@tcp [ 797.371918] LNet: Removed LNI 192.168.202.25@tcp [ 797.381358] LNet: Removed LNI 192.168.202.25@tcp2 [ 797.405414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 797.456566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 797.476804] LNet: Added LNI 192.168.202.25@tcp [8/256/0/180] [ 797.480275] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 797.485881] LNet: Added LNI 192.168.202.25@tcp2 [8/256/0/180] [ 803.526303] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 17:17:51 (1786828671) [ 803.731705] LNet: 45597:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 803.736903] LNetError: 45597:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 803.742648] LNetError: 45597:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 803.748295] LNet: Removed LNI 10.1.2.3@tcp [ 803.750681] LNet: Removed LNI 192.168.202.25@tcp [ 803.761199] LNet: Removed LNI 192.168.202.25@tcp2 [ 804.344825] Key type .llcrypt unregistered [ 804.346212] Key type ._llcrypt unregistered [ 810.375200] Key type ._llcrypt registered [ 810.380775] Key type .llcrypt registered [ 810.593269] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 810.607853] alg: No test for adler32 (adler32-zlib) [ 811.544729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 811.587408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 811.606229] LNet: Added LNI 192.168.202.25@tcp [8/256/0/180] [ 812.393223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 812.410859] LNet: 45995:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 812.415284] LNetError: 45995:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 812.426912] LNet: Removed LNI 192.168.202.25@tcp [ 812.443421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 812.503974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 812.538033] LNet: Added LNI 192.168.202.25@tcp [8/256/0/180] [ 813.088252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 813.112802] LNet: 46025:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 813.116542] LNetError: 46025:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 813.130979] LNet: Removed LNI 192.168.202.25@tcp [ 813.143575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 813.192277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 813.206029] LNet: Added LNI 192.168.202.25@tcp [8/256/0/180] [ 813.636639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 813.661980] LNet: Removed LNI 192.168.202.25@tcp [ 813.692354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 813.745098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 813.780653] LNet: Added LNI 192.168.202.25@tcp [8/256/0/180] [ 814.211394] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 814.324297] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 814.340266] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 814.347931] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 816.761709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 816.788726] LNet: 46132:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 816.795234] LNet: 46132:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 816.807081] LNetError: 46132:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 816.812120] LNetError: 46132:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 816.834735] LNet: Removed LNI 192.168.202.25@tcp [ 816.867718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 816.993652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 817.028954] LNet: Added LNI 192.168.202.25@tcp [8/256/0/180] [ 823.169890] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 17:18:10 (1786828690) [ 823.395581] LNet: 46729:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 823.402600] LNetError: 46729:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 823.418891] LNet: Removed LNI 192.168.202.25@tcp [ 823.963942] Key type .llcrypt unregistered [ 823.965863] Key type ._llcrypt unregistered [ 828.879305] Key type ._llcrypt registered [ 828.882483] Key type .llcrypt registered [ 829.036946] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 829.046485] alg: No test for adler32 (adler32-zlib) [ 829.962663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 830.027603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 830.074844] LNet: Added LNI 192.168.202.25@tcp [8/256/0/180] [ 830.773577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 830.786483] LNet: 47132:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 830.790847] LNetError: 47132:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 830.803319] LNet: Removed LNI 192.168.202.25@tcp [ 830.821628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 830.874658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 830.899084] LNet: Added LNI 192.168.202.25@tcp [8/256/0/180] [ 838.000598] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 17:18:25 (1786828705) [ 838.166682] LNet: 47716:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 838.177219] LNetError: 47716:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 838.206852] LNet: Removed LNI 192.168.202.25@tcp [ 839.099766] Key type .llcrypt unregistered [ 839.102142] Key type ._llcrypt unregistered [ 845.258951] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 845.372691] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 845.380210] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 845.393123] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 848.300848] Key type ._llcrypt registered [ 848.306605] Key type .llcrypt registered [ 848.532189] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 848.546887] alg: No test for adler32 (adler32-zlib) [ 849.543921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 849.595929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 849.620646] LNet: Added LNI 192.168.202.25@tcp [8/256/0/180] [ 850.397420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 850.413660] LNet: 48178:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 850.420307] LNetError: 48178:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 850.431604] LNet: Removed LNI 192.168.202.25@tcp [ 850.451373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 850.534901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 850.561460] LNet: Added LNI 192.168.202.25@tcp [8/256/0/180] [ 856.808761] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 17:18:44 (1786828724) [ 856.937556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 856.969584] LNet: 48767:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 856.976881] LNetError: 48767:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 857.002291] LNet: Removed LNI 192.168.202.25@tcp [ 857.042524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 857.192801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 863.263655] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 17:18:50 (1786828730) [ 863.534547] LNet: 49359:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 864.387663] Key type .llcrypt unregistered [ 864.390448] Key type ._llcrypt unregistered [ 871.166605] Key type ._llcrypt registered [ 871.175990] Key type .llcrypt registered [ 871.447842] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 871.466512] alg: No test for adler32 (adler32-zlib) [ 872.388603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 872.476680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 872.518659] LNet: Added LNI 192.168.202.25@tcp [8/256/0/180] [ 873.502567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 873.521515] LNet: 49760:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 873.528893] LNetError: 49760:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 873.540689] LNet: Removed LNI 192.168.202.25@tcp [ 873.556700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 873.636732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 873.666949] LNet: Added LNI 192.168.202.25@tcp [8/256/0/180] [ 882.051366] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 17:19:08 (1786828748) [ 882.190057] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 882.224480] LNet: 50346:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 882.240714] LNetError: 50346:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 882.287962] LNet: Removed LNI 192.168.202.25@tcp [ 882.339205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 882.479940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 889.811497] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 17:19:16 (1786828756) [ 891.564793] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 893.462160] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 17:19:20 (1786828760) [ 895.335698] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 897.074829] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 17:19:24 (1786828764) [ 899.105104] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 900.846281] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 17:19:27 (1786828767) [ 902.893873] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 905.299844] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 17:19:31 (1786828771) [ 907.170101] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 909.177807] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing ========================================================== 17:19:36 (1786828776) [ 909.257358] LNet: 52399:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 910.030271] Key type .llcrypt unregistered [ 910.035038] Key type ._llcrypt unregistered [ 917.388399] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 917.497243] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 917.509136] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 917.512788] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 919.115248] Key type ._llcrypt registered [ 919.118748] Key type .llcrypt registered [ 919.335970] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 919.356826] alg: No test for adler32 (adler32-zlib) [ 920.367668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 920.463968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 920.505092] LNet: Added LNI 192.168.202.25@tcp [8/512/0/180] [ 920.515423] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 920.518353] LNet: 52724:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 920.638677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 920.667985] LNet: 52743:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 920.678537] LNetError: 52743:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 920.690694] LNet: Removed LNI 10.1.2.3@tcp [ 920.702108] LNetError: 52743:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 920.718880] LNet: Removed LNI 192.168.202.25@tcp [ 920.775676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 920.854491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 920.887200] LNet: Added LNI 192.168.202.25@tcp [8/256/0/180] [ 920.984765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 921.016835] LNet: 52767:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 921.025272] LNetError: 52767:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 921.039817] LNet: Removed LNI 192.168.202.25@tcp [ 921.064594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 921.124512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 921.144050] LNet: Added LNI 192.168.202.25@tcp [8/256/0/180] [ 921.149318] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 921.154285] LNet: 52776:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 921.215127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 921.236854] LNet: 52794:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 921.242826] LNetError: 52794:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 921.252719] LNet: Removed LNI 10.1.2.3@tcp [ 921.255019] LNetError: 52794:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 921.266808] LNet: Removed LNI 192.168.202.25@tcp [ 921.287984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 921.347880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 921.365563] LNet: Added LNI 192.168.202.25@tcp [8/256/0/180] [ 921.372188] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 921.376375] LNet: 52802:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 921.623979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 921.647966] LNet: 52828:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 921.655181] LNetError: 52828:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 921.662795] LNet: Removed LNI 10.1.2.3@tcp [ 921.667389] LNetError: 52828:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 921.678213] LNet: Removed LNI 192.168.202.25@tcp [ 921.714669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 921.837459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 921.865574] LNet: Added LNI 192.168.202.25@tcp [8/256/0/182] [ 921.875716] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 922.014782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 922.046469] LNet: 52856:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 922.054638] LNetError: 52856:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 922.057720] LNet: Removed LNI 10.1.2.3@tcp [ 922.060396] LNetError: 52856:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 922.082812] LNet: Removed LNI 192.168.202.25@tcp [ 922.118175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 922.189619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 922.219968] LNet: Added LNI 192.168.202.25@tcp [8/256/0/182] [ 922.224093] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 922.227250] LNet: 52864:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 922.320810] LNet: Removed LNI 10.1.2.3@tcp [ 922.329821] LNet: Removed LNI 192.168.202.25@tcp [ 922.979238] Key type .llcrypt unregistered [ 922.983715] Key type ._llcrypt unregistered [ 935.459147] Key type ._llcrypt registered [ 935.463638] Key type .llcrypt registered [ 936.959343] Lustre: DEBUG MARKER: == sanity-lnet test 180: FastReg map error should not leak descriptors ========================================================== 17:20:03 (1786828803) [ 938.154929] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_180 Need o2ib NETTYPE [ 939.601912] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 17:20:06 (1786828806) [ 940.024152] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 940.065471] alg: No test for adler32 (adler32-zlib) [ 941.110905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 941.251469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 941.288983] LNet: Added LNI 192.168.202.25@tcp [8/256/0/180] [ 942.064810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.202.25 [ 942.089843] LNet: 54133:0:(config.c:1738:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 942.099556] LNet: Removed LNI 192.168.202.25@tcp [ 949.864185] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 17:20:16 (1786828816) [ 950.014246] LNet: 54694:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 950.029924] LNetError: 54694:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 950.055613] LNet: Removed LNI 192.168.202.25@tcp [ 950.835913] Key type .llcrypt unregistered [ 950.837983] Key type ._llcrypt unregistered [ 957.130690] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 957.399776] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 960.328913] Key type ._llcrypt registered [ 960.335810] Key type .llcrypt registered [ 960.554593] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 960.596462] alg: No test for adler32 (adler32-zlib) [ 961.926829] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 968.964918] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 17:20:35 (1786828835) [ 969.087765] LNet: 55612:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 969.109232] LNetError: 55612:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 969.146277] LNet: Removed LNI 10.1.2.3@tcp [ 969.920259] Key type .llcrypt unregistered [ 969.923649] Key type ._llcrypt unregistered [ 975.958355] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 976.286175] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 976.294420] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 976.303978] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 979.235796] Key type ._llcrypt registered [ 979.238225] Key type .llcrypt registered [ 979.411731] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 979.431758] alg: No test for adler32 (adler32-zlib) [ 980.542904] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 988.641950] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 17:20:55 (1786828855) [ 990.237141] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 991.789262] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 17:20:58 (1786828858) [ 991.910176] LNet: 56800:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 991.914200] LNetError: 56800:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 991.942175] LNet: Removed LNI 10.1.2.3@tcp [ 992.632868] Key type .llcrypt unregistered [ 992.639350] Key type ._llcrypt unregistered [ 998.786358] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 999.020971] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 999.023433] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 999.025395] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1001.811876] Key type ._llcrypt registered [ 1001.818479] Key type .llcrypt registered [ 1002.152184] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1002.169717] alg: No test for adler32 (adler32-zlib) [ 1003.124218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1003.216440] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1012.076195] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 17:21:19 (1786828879) [ 1012.518797] LNet: 58042:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1012.528799] LNetError: 58042:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1012.544197] LNet: Removed LNI 10.1.2.3@tcp [ 1013.246777] Key type .llcrypt unregistered [ 1013.252215] Key type ._llcrypt unregistered [ 1021.302890] Key type ._llcrypt registered [ 1021.304974] Key type .llcrypt registered [ 1021.643393] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1021.662858] alg: No test for adler32 (adler32-zlib) [ 1022.728969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1022.749406] LNet: Added LNI 192.168.202.25@tcp [8/256/0/180] [ 1031.382780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.125@tcp [ 1032.175978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.25@tcp -d 192.168.202.125@tcp -m GET -e local_interrupt -r 1 [ 1032.237546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.25@tcp -d 192.168.202.25@tcp -m GET -e local_interrupt -r 1 [ 1032.306533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.125@tcp [ 1032.333499] LNet: There was an unexpected network error while writing to 192.168.202.125: rc = -22 [ 1032.634484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1032.662028] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1032.690940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1032.723293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1032.760968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1032.790919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1032.838373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1032.857315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1032.867179] LNet: 58484:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1032.879501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1032.913294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1032.956426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1032.991550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1032.992176] LNet: 1 local NIs in recovery (showing 1): 192.168.202.25@tcp [ 1033.023336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1033.055878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1033.075918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1033.666900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.25@tcp -d 192.168.202.125@tcp -m GET -e local_dropped -r 1 [ 1033.709712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.25@tcp -d 192.168.202.25@tcp -m GET -e local_dropped -r 1 [ 1033.742274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.125@tcp [ 1033.761995] LNet: There was an unexpected network error while writing to 192.168.202.125: rc = -22 [ 1034.013758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1034.058723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1034.110413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1034.140765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1034.194924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1034.255782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1034.344818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1034.400706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1034.432697] LNet: 58593:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1034.464734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1034.511360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1034.553430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1034.614166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1034.662576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1034.732210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1034.790992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1035.440218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.25@tcp -d 192.168.202.125@tcp -m GET -e local_aborted -r 1 [ 1035.477292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.25@tcp -d 192.168.202.25@tcp -m GET -e local_aborted -r 1 [ 1035.511777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.125@tcp [ 1035.528836] LNet: There was an unexpected network error while writing to 192.168.202.125: rc = -22 [ 1035.532751] LNet: Skipped 1 previous similar message [ 1035.767629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1035.808096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1035.840679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1035.870608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1035.902411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1035.932158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1035.962720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1035.992943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1036.014885] LNet: 58702:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1036.038870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1036.096478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1036.124683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1036.160692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1036.201906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1036.259056] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1036.303413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1036.912598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.25@tcp -d 192.168.202.125@tcp -m GET -e local_no_route -r 1 [ 1036.949550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.25@tcp -d 192.168.202.25@tcp -m GET -e local_no_route -r 1 [ 1037.008835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.125@tcp [ 1037.340639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1037.382526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1037.426913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1037.466042] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1037.505956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1037.545955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1037.599498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1037.649475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1037.718184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1037.780284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1037.827441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1037.870783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1037.907293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1037.941351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1037.970572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1038.639343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.25@tcp -d 192.168.202.125@tcp -m GET -e local_timeout -r 1 [ 1038.676276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.25@tcp -d 192.168.202.25@tcp -m GET -e local_timeout -r 1 [ 1038.753936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.125@tcp [ 1038.775627] LNet: There was an unexpected network error while writing to 192.168.202.125: rc = -22 [ 1038.779546] LNet: Skipped 2 previous similar messages [ 1039.064934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1039.094443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1039.117849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1039.154241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1039.189349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1039.221494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1039.285869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1039.350312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1039.374910] LNet: 58921:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1039.386282] LNet: 58921:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1039.430456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1039.502954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1039.522333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1039.545709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1039.563828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1039.583693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1039.617702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1040.097930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.25@tcp -d 192.168.202.125@tcp -m GET -e local_error -r 1 [ 1040.128930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.25@tcp -d 192.168.202.25@tcp -m GET -e local_error -r 1 [ 1040.164491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.125@tcp [ 1040.407202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1040.435628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1040.463743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1040.491253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1040.525463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1040.549842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1040.576380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1040.600963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1040.629759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1040.661968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1040.692788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1040.720913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1040.747910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1040.774290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1040.804969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1041.028672] LNet: 59075:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1041.035173] LNetError: 59075:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1042.110494] LNet: Removed LNI 192.168.202.25@tcp [ 1042.538882] Key type .llcrypt unregistered [ 1042.542864] Key type ._llcrypt unregistered [ 1060.846068] Key type ._llcrypt registered [ 1060.851554] Key type .llcrypt registered [ 1062.760477] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 17:22:09 (1786828929) [ 1064.137420] Key type .llcrypt unregistered [ 1064.143502] Key type ._llcrypt unregistered [ 1072.173825] Key type ._llcrypt registered [ 1072.176386] Key type .llcrypt registered [ 1072.594630] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1072.610716] alg: No test for adler32 (adler32-zlib) [ 1073.563237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1073.612463] LNet: Added LNI 192.168.202.25@tcp [8/256/0/180] [ 1082.154627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.125@tcp [ 1094.275836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1094.341174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1094.365737] LNet: Added LNI 192.168.202.25@tcp1 [8/256/0/180] [ 1095.816973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.25@tcp -d 192.168.202.125@tcp -m GET -e local_interrupt -r 1 [ 1095.875788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.25@tcp -d 192.168.202.125@tcp1 -m GET -e local_interrupt -r 1 [ 1095.936480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.25@tcp -d 192.168.202.25@tcp -m GET -e local_interrupt -r 1 [ 1096.005757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.25@tcp -d 192.168.202.25@tcp1 -m GET -e local_interrupt -r 1 [ 1096.067672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.25@tcp1 -d 192.168.202.125@tcp -m GET -e local_interrupt -r 1 [ 1096.125901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.25@tcp1 -d 192.168.202.125@tcp1 -m GET -e local_interrupt -r 1 [ 1096.227984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.25@tcp1 -d 192.168.202.25@tcp -m GET -e local_interrupt -r 1 [ 1096.265336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.25@tcp1 -d 192.168.202.25@tcp1 -m GET -e local_interrupt -r 1 [ 1096.302728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.125@tcp [ 1096.315751] LNet: There was an unexpected network error while writing to 192.168.202.125: rc = -22 [ 1096.323783] LNet: 1 local NIs in recovery (showing 1): 192.168.202.25@tcp [ 1096.520805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1096.567749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1096.596854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1096.646336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1096.681801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1096.711990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1096.743401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1096.774449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1096.791416] LNet: 60547:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1096.809105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1096.836977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1096.876507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1096.909371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1096.970654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1097.007891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1097.049407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1097.550969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.25@tcp -d 192.168.202.125@tcp -m GET -e local_dropped -r 1 [ 1097.585412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.25@tcp -d 192.168.202.125@tcp1 -m GET -e local_dropped -r 1 [ 1097.618770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.25@tcp -d 192.168.202.25@tcp -m GET -e local_dropped -r 1 [ 1097.646352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.25@tcp -d 192.168.202.25@tcp1 -m GET -e local_dropped -r 1 [ 1097.702203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.25@tcp1 -d 192.168.202.125@tcp -m GET -e local_dropped -r 1 [ 1097.743765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.25@tcp1 -d 192.168.202.125@tcp1 -m GET -e local_dropped -r 1 [ 1097.772402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.25@tcp1 -d 192.168.202.25@tcp -m GET -e local_dropped -r 1 [ 1097.819285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.25@tcp1 -d 192.168.202.25@tcp1 -m GET -e local_dropped -r 1 [ 1097.874781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.125@tcp [ 1097.907333] LNet: There was an unexpected network error while writing to 192.168.202.125: rc = -22 [ 1097.935371] LNet: Skipped 2 previous similar messages [ 1098.251280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1098.281649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1098.319857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1098.356378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1098.387562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1098.406543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1098.425692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1098.457356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1098.480318] LNet: 60668:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1098.518951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1098.585839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1098.635576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1098.693505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1098.740851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1098.769893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1098.805893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1099.463937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.25@tcp -d 192.168.202.125@tcp -m GET -e local_aborted -r 1 [ 1099.492759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.25@tcp -d 192.168.202.125@tcp1 -m GET -e local_aborted -r 1 [ 1099.521709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.25@tcp -d 192.168.202.25@tcp -m GET -e local_aborted -r 1 [ 1099.547576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.25@tcp -d 192.168.202.25@tcp1 -m GET -e local_aborted -r 1 [ 1099.583635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.25@tcp1 -d 192.168.202.125@tcp -m GET -e local_aborted -r 1 [ 1099.611909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.25@tcp1 -d 192.168.202.125@tcp1 -m GET -e local_aborted -r 1 [ 1099.638396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.25@tcp1 -d 192.168.202.25@tcp -m GET -e local_aborted -r 1 [ 1099.666350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.25@tcp1 -d 192.168.202.25@tcp1 -m GET -e local_aborted -r 1 [ 1099.693255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.125@tcp [ 1099.714944] LNet: There was an unexpected network error while writing to 192.168.202.125: rc = -22 [ 1099.719784] LNet: Skipped 2 previous similar messages [ 1099.946710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1099.985862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1100.021551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1100.060903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1100.113607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1100.181033] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1100.221225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1100.248773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1100.267529] LNet: 60789:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1100.284599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1100.324840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1100.372717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1100.401175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1100.448707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1100.503418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1100.566534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1101.189904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.25@tcp -d 192.168.202.125@tcp -m GET -e local_no_route -r 1 [ 1101.216223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.25@tcp -d 192.168.202.125@tcp1 -m GET -e local_no_route -r 1 [ 1101.246614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.25@tcp -d 192.168.202.25@tcp -m GET -e local_no_route -r 1 [ 1101.282852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.25@tcp -d 192.168.202.25@tcp1 -m GET -e local_no_route -r 1 [ 1101.315664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.25@tcp1 -d 192.168.202.125@tcp -m GET -e local_no_route -r 1 [ 1101.344767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.25@tcp1 -d 192.168.202.125@tcp1 -m GET -e local_no_route -r 1 [ 1101.365392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.25@tcp1 -d 192.168.202.25@tcp -m GET -e local_no_route -r 1 [ 1101.391897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.25@tcp1 -d 192.168.202.25@tcp1 -m GET -e local_no_route -r 1 [ 1101.427434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.125@tcp [ 1101.684911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1101.743866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1101.777468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1101.812866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1101.861301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1101.913812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1101.972390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1102.043879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1102.077985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1102.119865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1102.168687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1102.194802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1102.215852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1102.261773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1102.298527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1102.958426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.25@tcp -d 192.168.202.125@tcp -m GET -e local_timeout -r 1 [ 1103.005261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.25@tcp -d 192.168.202.125@tcp1 -m GET -e local_timeout -r 1 [ 1103.061196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.25@tcp -d 192.168.202.25@tcp -m GET -e local_timeout -r 1 [ 1103.106836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.25@tcp -d 192.168.202.25@tcp1 -m GET -e local_timeout -r 1 [ 1103.192178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.25@tcp1 -d 192.168.202.125@tcp -m GET -e local_timeout -r 1 [ 1103.240826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.25@tcp1 -d 192.168.202.125@tcp1 -m GET -e local_timeout -r 1 [ 1103.298638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.25@tcp1 -d 192.168.202.25@tcp -m GET -e local_timeout -r 1 [ 1103.380364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.25@tcp1 -d 192.168.202.25@tcp1 -m GET -e local_timeout -r 1 [ 1103.434788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.125@tcp [ 1103.483772] LNet: There was an unexpected network error while writing to 192.168.202.125: rc = -22 [ 1103.489764] LNet: Skipped 5 previous similar messages [ 1103.718884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1103.748230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1103.774787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1103.834373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1103.897573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1103.931875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1103.982917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1104.019733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1104.053925] LNet: 61032:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1104.064824] LNet: 61032:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1104.106253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1104.155482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1104.214608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1104.239463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1104.260859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1104.281466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1104.303266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1104.808301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.25@tcp -d 192.168.202.125@tcp -m GET -e local_error -r 1 [ 1104.826673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.25@tcp -d 192.168.202.125@tcp1 -m GET -e local_error -r 1 [ 1104.850349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.25@tcp -d 192.168.202.25@tcp -m GET -e local_error -r 1 [ 1104.877925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.25@tcp -d 192.168.202.25@tcp1 -m GET -e local_error -r 1 [ 1104.905615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.25@tcp1 -d 192.168.202.125@tcp -m GET -e local_error -r 1 [ 1104.931711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.25@tcp1 -d 192.168.202.125@tcp1 -m GET -e local_error -r 1 [ 1104.978343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.25@tcp1 -d 192.168.202.25@tcp -m GET -e local_error -r 1 [ 1105.042218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.25@tcp1 -d 192.168.202.25@tcp1 -m GET -e local_error -r 1 [ 1105.097727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.125@tcp [ 1105.407890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1105.480747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1105.552648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1105.601809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1105.639781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1105.682962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1105.714616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1105.759387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1105.799774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1105.859863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1105.909466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1105.951645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1105.971651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1105.992789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1106.022370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1111.634374] LNet: 61233:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1111.639776] LNetError: 61233:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1111.644218] LNet: Removed LNI 192.168.202.25@tcp [ 1111.663540] LNet: Removed LNI 192.168.202.25@tcp1 [ 1112.089604] Key type .llcrypt unregistered [ 1112.092103] Key type ._llcrypt unregistered [ 1129.495276] Key type ._llcrypt registered [ 1129.497229] Key type .llcrypt registered [ 1131.350753] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 17:23:17 (1786828997) [ 1132.393375] Key type .llcrypt unregistered [ 1132.398986] Key type ._llcrypt unregistered [ 1139.549874] Key type ._llcrypt registered [ 1139.553028] Key type .llcrypt registered [ 1139.829318] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1139.844852] alg: No test for adler32 (adler32-zlib) [ 1140.775472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1140.815636] LNet: Added LNI 192.168.202.25@tcp [8/256/0/180] [ 1147.979975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.125@tcp [ 1148.888815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.25@tcp -d 192.168.202.125@tcp -m GET -e remote_dropped -r 1 [ 1148.921787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.25@tcp -d 192.168.202.25@tcp -m GET -e remote_dropped -r 1 [ 1148.968933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.125@tcp [ 1148.997205] LNet: There was an unexpected network error while writing to 192.168.202.125: rc = -22 [ 1149.358408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1149.391987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1149.430588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1149.474158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1149.543766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1149.602178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1149.640975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1149.702532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1149.736118] LNet: 62487:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1149.763872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1149.817483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1149.858314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1149.890373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1149.968219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1150.016564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1150.055948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1150.644719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.25@tcp -d 192.168.202.125@tcp -m GET -e remote_error -r 1 [ 1150.710677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.25@tcp -d 192.168.202.25@tcp -m GET -e remote_error -r 1 [ 1150.788843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.125@tcp [ 1150.839085] LNet: There was an unexpected network error while writing to 192.168.202.125: rc = -22 [ 1151.207392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1151.268302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1151.341686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1151.389368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1151.474313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1151.525656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1151.564478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1151.607458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1151.637192] LNet: 62597:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1151.655677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1151.686673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1151.711300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1151.735906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1151.787929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1151.826289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1151.858983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1152.419484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.25@tcp -d 192.168.202.125@tcp -m GET -e remote_timeout -r 1 [ 1152.459322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.25@tcp -d 192.168.202.25@tcp -m GET -e remote_timeout -r 1 [ 1152.505643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.125@tcp [ 1152.530243] LNet: There was an unexpected network error while writing to 192.168.202.125: rc = -22 [ 1152.792357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1152.835100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1152.879171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1152.920146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1152.962384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1153.013740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1153.077417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1153.131949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1153.159243] LNet: 62706:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1153.186363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1153.219991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1153.261627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1153.318484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1153.354500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1153.401785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1153.454942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1153.932991] LNet: 62751:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1153.943961] LNetError: 62751:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1153.969029] LNet: Removed LNI 192.168.202.25@tcp [ 1154.562265] Key type .llcrypt unregistered [ 1154.570632] Key type ._llcrypt unregistered [ 1170.947490] Key type ._llcrypt registered [ 1170.949307] Key type .llcrypt registered [ 1172.461181] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 17:23:59 (1786829039) [ 1173.585491] Key type .llcrypt unregistered [ 1173.588173] Key type ._llcrypt unregistered [ 1180.572483] Key type ._llcrypt registered [ 1180.575574] Key type .llcrypt registered [ 1180.892161] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1180.950413] alg: No test for adler32 (adler32-zlib) [ 1181.871891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1181.915721] LNet: Added LNI 192.168.202.25@tcp [8/256/0/180] [ 1189.058899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.125@tcp [ 1199.564262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1199.638919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1199.668081] LNet: Added LNI 192.168.202.25@tcp1 [8/256/0/180] [ 1200.980972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.25@tcp -d 192.168.202.125@tcp -m GET -e remote_dropped -r 1 [ 1201.005386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.25@tcp -d 192.168.202.125@tcp1 -m GET -e remote_dropped -r 1 [ 1201.054959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.25@tcp -d 192.168.202.25@tcp -m GET -e remote_dropped -r 1 [ 1201.092104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.25@tcp -d 192.168.202.25@tcp1 -m GET -e remote_dropped -r 1 [ 1201.139476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.25@tcp1 -d 192.168.202.125@tcp -m GET -e remote_dropped -r 1 [ 1201.183915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.25@tcp1 -d 192.168.202.125@tcp1 -m GET -e remote_dropped -r 1 [ 1201.216427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.25@tcp1 -d 192.168.202.25@tcp -m GET -e remote_dropped -r 1 [ 1201.259707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.25@tcp1 -d 192.168.202.25@tcp1 -m GET -e remote_dropped -r 1 [ 1201.295800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.125@tcp [ 1201.311473] LNet: There was an unexpected network error while writing to 192.168.202.125: rc = -22 [ 1201.316847] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.125@tcp [ 1201.534800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1201.559370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1201.588828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1201.620988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1201.659299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1201.687556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1201.721461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1201.765498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1201.790794] LNet: 64224:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1201.815784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1201.864286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1201.925932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1201.960413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1202.000104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1202.036488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1202.075796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1202.144899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1202.820888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.25@tcp -d 192.168.202.125@tcp -m GET -e remote_error -r 1 [ 1202.854925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.25@tcp -d 192.168.202.125@tcp1 -m GET -e remote_error -r 1 [ 1202.900453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.25@tcp -d 192.168.202.25@tcp -m GET -e remote_error -r 1 [ 1202.932064] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.25@tcp -d 192.168.202.25@tcp1 -m GET -e remote_error -r 1 [ 1202.983329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.25@tcp1 -d 192.168.202.125@tcp -m GET -e remote_error -r 1 [ 1203.049087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.25@tcp1 -d 192.168.202.125@tcp1 -m GET -e remote_error -r 1 [ 1203.089497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.25@tcp1 -d 192.168.202.25@tcp -m GET -e remote_error -r 1 [ 1203.132300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.25@tcp1 -d 192.168.202.25@tcp1 -m GET -e remote_error -r 1 [ 1203.186980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.125@tcp [ 1203.229449] LNet: There was an unexpected network error while writing to 192.168.202.125: rc = -22 [ 1203.236964] LNet: Skipped 2 previous similar messages [ 1203.467789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1203.506698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1203.531330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1203.559382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1203.630816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1203.679953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1203.740956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1203.830904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1203.867911] LNet: 64348:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1203.907881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1203.943973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1203.985432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1204.063730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1204.099380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1204.160057] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1204.211963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1204.259691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1204.779496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.25@tcp -d 192.168.202.125@tcp -m GET -e remote_timeout -r 1 [ 1204.817996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.25@tcp -d 192.168.202.125@tcp1 -m GET -e remote_timeout -r 1 [ 1204.878939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.25@tcp -d 192.168.202.25@tcp -m GET -e remote_timeout -r 1 [ 1204.952722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.25@tcp -d 192.168.202.25@tcp1 -m GET -e remote_timeout -r 1 [ 1205.015423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.25@tcp1 -d 192.168.202.125@tcp -m GET -e remote_timeout -r 1 [ 1205.053424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.25@tcp1 -d 192.168.202.125@tcp1 -m GET -e remote_timeout -r 1 [ 1205.110305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.25@tcp1 -d 192.168.202.25@tcp -m GET -e remote_timeout -r 1 [ 1205.154645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.25@tcp1 -d 192.168.202.25@tcp1 -m GET -e remote_timeout -r 1 [ 1205.203676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.125@tcp [ 1205.223808] LNet: There was an unexpected network error while writing to 192.168.202.125: rc = -22 [ 1205.439078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1205.462369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1205.499299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1205.518812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1205.545474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1205.587400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1205.629592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1205.685910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1205.704239] LNet: 64471:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1205.725583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1205.786819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1205.851956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1205.889295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1205.924932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1205.958262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1206.010678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1206.061315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1210.954782] LNet: 64553:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1210.959716] LNetError: 64553:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1212.003657] LNet: Removed LNI 192.168.202.25@tcp [ 1212.012853] LNetError: 64553:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1212.049352] LNet: Removed LNI 192.168.202.25@tcp1 [ 1212.620501] Key type .llcrypt unregistered [ 1212.627631] Key type ._llcrypt unregistered [ 1230.742365] Key type ._llcrypt registered [ 1230.745840] Key type .llcrypt registered [ 1232.040696] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 17:24:59 (1786829099) [ 1233.593751] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 1235.966499] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 17:25:02 (1786829102) [ 1237.025437] Key type .llcrypt unregistered [ 1237.028790] Key type ._llcrypt unregistered [ 1244.202550] Key type ._llcrypt registered [ 1244.204972] Key type .llcrypt registered [ 1244.419392] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1244.437466] alg: No test for adler32 (adler32-zlib) [ 1245.364752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1245.394634] LNet: Added LNI 192.168.202.25@tcp [8/256/0/180] [ 1253.081572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.125@tcp [ 1254.122615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.25@tcp -d 192.168.202.125@tcp -m GET -e network_timeout -r 1 [ 1254.203874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.25@tcp -d 192.168.202.25@tcp -m GET -e network_timeout -r 1 [ 1254.265519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.125@tcp [ 1254.308519] LNet: There was an unexpected network error while writing to 192.168.202.125: rc = -22 [ 1254.596388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1254.625170] LNet: 1 local NIs in recovery (showing 1): 192.168.202.25@tcp [ 1254.640257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1254.689516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1254.732386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1254.769720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1254.817906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1254.845299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1254.896611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1254.917411] LNet: 66103:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1254.940626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1255.009831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1255.041724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1255.082275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1255.115295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1255.172420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1255.573467] LNet: 66146:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1255.577667] LNetError: 66146:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1255.587469] LNet: Removed LNI 192.168.202.25@tcp [ 1256.114618] Key type .llcrypt unregistered [ 1256.118364] Key type ._llcrypt unregistered [ 1276.646094] Key type ._llcrypt registered [ 1276.651642] Key type .llcrypt registered [ 1276.962870] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1276.993828] alg: No test for adler32 (adler32-zlib) [ 1277.963451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1277.989158] LNet: Added LNI 192.168.202.25@tcp [8/256/0/180] [ 1285.690716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.125@tcp [ 1294.652358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1294.715530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1294.745908] LNet: Added LNI 192.168.202.25@tcp1 [8/256/0/180] [ 1296.592877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.25@tcp -d 192.168.202.125@tcp -m GET -e network_timeout -r 1 [ 1296.645230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.25@tcp -d 192.168.202.125@tcp1 -m GET -e network_timeout -r 1 [ 1296.687726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.25@tcp -d 192.168.202.25@tcp -m GET -e network_timeout -r 1 [ 1296.719897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.25@tcp -d 192.168.202.25@tcp1 -m GET -e network_timeout -r 1 [ 1296.756518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.25@tcp1 -d 192.168.202.125@tcp -m GET -e network_timeout -r 1 [ 1296.801601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.25@tcp1 -d 192.168.202.125@tcp1 -m GET -e network_timeout -r 1 [ 1296.854680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.25@tcp1 -d 192.168.202.25@tcp -m GET -e network_timeout -r 1 [ 1296.927267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.25@tcp1 -d 192.168.202.25@tcp1 -m GET -e network_timeout -r 1 [ 1296.971038] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.125@tcp [ 1296.996571] LNet: There was an unexpected network error while writing to 192.168.202.125: rc = -22 [ 1297.333460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1297.372602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1297.428191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1297.453450] LNet: 1 local NIs in recovery (showing 1): 192.168.202.25@tcp [ 1297.458178] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.125@tcp [ 1297.501071] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1297.561407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1297.602667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1297.651981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1297.693269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1297.729910] LNet: 67031:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1297.745852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1297.772878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1297.804959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1297.850477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1297.913457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1297.978147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1302.094516] LNet: There was an unexpected network error while writing to 192.168.202.125: rc = -22 [ 1302.562299] LNet: 67109:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1302.567437] LNetError: 67109:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1303.584477] LNet: Removed LNI 192.168.202.25@tcp [ 1303.586798] LNetError: 67109:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1303.605117] LNet: Removed LNI 192.168.202.25@tcp1 [ 1303.968724] Key type .llcrypt unregistered [ 1303.971922] Key type ._llcrypt unregistered [ 1326.028368] Key type ._llcrypt registered [ 1326.030877] Key type .llcrypt registered [ 1328.220162] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 17:26:34 (1786829194) [ 1328.666820] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1328.687636] alg: No test for adler32 (adler32-zlib) [ 1329.671927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1329.735051] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1329.764956] LNet: Added LNI 192.168.202.25@tcp [8/256/0/180] [ 1330.897452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1330.935601] LNet: Added LNI 192.168.202.25@tcp1 [8/256/0/180] [ 1332.073607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.25@tcp [ 1332.161829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1332.211786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1332.236506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1332.272520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1332.314537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1332.369949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.202.25@tcp [ 1332.394971] LNetError: 68211:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.202.25@tcp to recovery [ 1332.768338] LNet: 1 local NIs in recovery (showing 1): 192.168.202.25@tcp [ 1332.770552] LNet: There was an unexpected network error while writing to 192.168.202.25: rc = -22 [ 1334.816516] LNet: There was an unexpected network error while writing to 192.168.202.25: rc = -22 [ 1335.879799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1338.912408] LNet: There was an unexpected network error while writing to 192.168.202.25: rc = -22 [ 1339.335635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1339.353141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1339.393347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1339.412816] LNet: 68318:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1339.418258] LNetError: 68318:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1339.424078] LNet: Removed LNI 192.168.202.25@tcp [ 1339.430936] LNet: Removed LNI 192.168.202.25@tcp1 [ 1339.448807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1339.516686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1339.548359] LNet: Added LNI 192.168.202.25@tcp [8/256/0/180] [ 1340.327484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1340.355227] LNet: Added LNI 192.168.202.25@tcp1 [8/256/0/180] [ 1341.688649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.25@tcp [ 1341.727764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1341.836200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1341.900785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1341.971455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1342.101455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1342.235701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1342.319361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.202.25@tcp [ 1342.348744] LNetError: 68591:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.202.25@tcp to recovery [ 1342.564936] LNet: There was an unexpected network error while writing to 192.168.202.25: rc = -22 [ 1345.020370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1348.705563] LNet: There was an unexpected network error while writing to 192.168.202.25: rc = -22 [ 1348.716072] LNet: Skipped 1 previous similar message [ 1353.590857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1353.639728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1353.697245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1362.996253] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 17:27:10 (1786829230) [ 1363.072290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1363.094561] LNet: 69294:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1363.101112] LNetError: 69294:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1363.105282] LNetError: 69294:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1364.129436] LNet: Removed LNI 192.168.202.25@tcp [ 1364.135890] LNetError: 69294:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1364.148292] LNet: Removed LNI 192.168.202.25@tcp1 [ 1364.177263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1364.267357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1364.297401] LNet: Added LNI 192.168.202.25@tcp [8/256/0/180] [ 1365.186248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1365.204996] LNet: Added LNI 192.168.202.25@tcp1 [8/256/0/180] [ 1366.256167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.25@tcp [ 1366.376135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1366.422595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1366.458901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1366.495471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1366.533369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1366.571402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.202.25@tcp --health 0 [ 1366.682376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1367.328309] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.25@tcp [ 1367.328854] LNet: There was an unexpected network error while writing to 192.168.202.25: rc = -22 [ 1367.343471] LNet: Skipped 1 previous similar message [ 1382.279017] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1382.481339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1382.509747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1382.527419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.202.25@tcp --health 500 [ 1382.590876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1389.341441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1389.438441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1389.453906] LNet: 69792:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1389.460720] LNetError: 69792:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1389.467445] LNet: Removed LNI 192.168.202.25@tcp [ 1389.478075] LNet: Removed LNI 192.168.202.25@tcp1 [ 1389.544707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1389.626320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1389.656853] LNet: Added LNI 192.168.202.25@tcp [8/256/0/180] [ 1390.352102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1390.375404] LNet: Added LNI 192.168.202.25@tcp1 [8/256/0/180] [ 1391.314638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.25@tcp [ 1391.345978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1391.397874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1391.418896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1391.441474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1391.463358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1391.497779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1391.520981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.202.25@tcp --health 0 [ 1392.673409] LNet: There was an unexpected network error while writing to 192.168.202.25: rc = -22 [ 1392.690362] LNet: Skipped 3 previous similar messages [ 1392.977191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1403.308984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1403.333180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1403.394296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1410.676343] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 17:27:57 (1786829277) [ 1411.221524] LNet: 70839:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1411.226748] LNetError: 70839:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1411.232639] LNetError: 70839:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1411.238926] LNet: Removed LNI 192.168.202.25@tcp [ 1411.247636] LNet: Removed LNI 192.168.202.25@tcp1 [ 1411.798760] Key type .llcrypt unregistered [ 1411.800860] Key type ._llcrypt unregistered [ 1419.031316] Key type ._llcrypt registered [ 1419.035931] Key type .llcrypt registered [ 1419.221333] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1419.228713] alg: No test for adler32 (adler32-zlib) [ 1420.218393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1420.243591] LNet: Added LNI 192.168.202.25@tcp [8/256/0/180] [ 1431.169501] Lustre: DEBUG MARKER: Initial discovery [ 1431.241587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.202.125@tcp [ 1434.168603] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1434.286693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1434.383613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.202.125@tcp [ 1434.430474] LNet: There was an unexpected network error while writing to 192.168.202.125: rc = -22 [ 1434.457959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1434.537602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.202.125@tcp -r 1 -m GET -l 3 [ 1434.592124] LNet: 1 local NIs in recovery (showing 1): 192.168.202.25@tcp [ 1434.620391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.202.125@tcp -r 1 -m GET -e local_error [ 1434.668531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.202.125@tcp -r 1 -m PUT -l 6 [ 1436.910771] Lustre: DEBUG MARKER: Force oleg225-client.virtnet to discover 192.168.202.125@tcp (in background) [ 1437.144846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.202.125@tcp [ 1439.245753] Lustre: DEBUG MARKER: Force oleg225-server to discover 192.168.202.25@tcp [ 1440.224427] LNet: There was an unexpected network error while writing to 192.168.202.125: rc = -22 [ 1442.839788] Lustre: DEBUG MARKER: Wait for 71602 [ 1443.633938] LNet: There was an unexpected network error while writing to 192.168.202.125: rc = -22 [ 1445.279386] Lustre: DEBUG MARKER: Finished wait on 71602 [ 1445.386186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay del -a [ 1445.492875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1445.897452] LNet: 72062:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1445.910504] LNetError: 72062:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1446.958556] LNet: Removed LNI 192.168.202.25@tcp [ 1447.479737] Key type .llcrypt unregistered [ 1447.482576] Key type ._llcrypt unregistered [ 1464.778557] Key type ._llcrypt registered [ 1464.781607] Key type .llcrypt registered [ 1466.001829] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 17:28:53 (1786829333) [ 1466.305489] Key type .llcrypt unregistered [ 1466.307768] Key type ._llcrypt unregistered [ 1472.206362] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1472.493750] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1472.500567] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1472.510938] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1474.395893] Key type ._llcrypt registered [ 1474.397955] Key type .llcrypt registered [ 1474.640493] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1474.659588] alg: No test for adler32 (adler32-zlib) [ 1475.794785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1475.905301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1475.940898] LNet: Added LNI 192.168.202.25@tcp [8/256/0/180] [ 1477.216184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1477.243741] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1487.664065] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 17:29:14 (1786829354) [ 1487.838221] LNet: 74129:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1487.846551] LNetError: 74129:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1487.854172] LNet: Removed LNI 10.1.2.3@tcp [ 1487.872880] LNet: Removed LNI 192.168.202.25@tcp [ 1488.992394] Key type .llcrypt unregistered [ 1488.993993] Key type ._llcrypt unregistered [ 1496.669727] Key type ._llcrypt registered [ 1496.671092] Key type .llcrypt registered [ 1496.858794] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1496.869432] alg: No test for adler32 (adler32-zlib) [ 1497.719916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1497.758160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1497.777401] LNet: Added LNI 192.168.202.25@tcp [8/256/0/180] [ 1498.218756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1498.237928] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1499.155508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.202.25@tcp [ 1500.192183] LNetError: Unexpected error -22 connecting to 192.168.202.25@tcp at host 192.168.202.25:988 [ 1506.463552] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 17:29:33 (1786829373) [ 1506.711099] LNet: 75272:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1506.718271] LNetError: 75272:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1506.723767] LNet: Removed LNI 10.1.2.3@tcp [ 1506.737442] LNet: Removed LNI 192.168.202.25@tcp [ 1507.527462] Key type .llcrypt unregistered [ 1507.532764] Key type ._llcrypt unregistered [ 1516.300865] Key type ._llcrypt registered [ 1516.303944] Key type .llcrypt registered [ 1516.626356] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1516.665725] alg: No test for adler32 (adler32-zlib) [ 1517.680270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1517.802973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1517.863275] LNet: Added LNI 192.168.202.25@tcp1 [8/256/0/180] [ 1518.910620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1518.937482] LNet: Added LNI 192.168.202.25@tcp2 [8/256/0/180] [ 1519.752705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.202.25@tcp1 --nid 192.168.202.25@tcp2 [ 1524.670950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.25@tcp1 192.168.202.25@tcp2 [ 1524.690744] LNetError: 76119:0:(lib-move.c:2101:lnet_handle_find_routed_path()) Unable to send message from 192.168.202.25@tcp1 to 192.168.202.25@tcp2 - Route table may be misconfigured [ 1524.722603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.25@tcp2 192.168.202.25@tcp1 [ 1532.423150] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 17:29:59 (1786829399) [ 1532.551495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1532.576625] LNet: 76686:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1532.581706] LNetError: 76686:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1533.601039] LNet: Removed LNI 192.168.202.25@tcp1 [ 1533.604473] LNetError: 76686:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1534.632607] LNet: Removed LNI 192.168.202.25@tcp2 [ 1534.652802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1534.701585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1534.722276] LNet: Added LNI 192.168.202.25@tcp [8/256/0/180] [ 1535.276906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1535.294523] LNet: Added LNI 192.168.202.25@tcp1 [8/256/0/180] [ 1535.972693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.25@tcp [ 1536.008350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.25@tcp 192.168.202.25@tcp [ 1536.044695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.25@tcp1 192.168.202.25@tcp1 [ 1536.084764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.202.25@tcp -d 192.168.202.25@tcp -e network_timeout [ 1536.117105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.202.25@tcp -d 192.168.202.25@tcp1 -e network_timeout [ 1536.180701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.202.25@tcp1 -d 192.168.202.25@tcp -e network_timeout [ 1536.224726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.202.25@tcp1 -d 192.168.202.25@tcp1 -e network_timeout [ 1536.268727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.25@tcp [ 1536.313146] LNet: There was an unexpected network error while writing to 192.168.202.25: rc = -22 [ 1536.337101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1536.437541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1536.551526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1536.736466] LNet: 1 local NIs in recovery (showing 1): 192.168.202.25@tcp [ 1542.960191] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 17:30:09 (1786829409) [ 1543.213503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1543.239777] LNet: 77555:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1543.247433] LNetError: 77555:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1543.258102] LNet: Removed LNI 192.168.202.25@tcp [ 1544.300084] LNet: Removed LNI 192.168.202.25@tcp1 [ 1544.341849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1544.465264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1544.881568] LNet: 77596:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1545.300576] Key type .llcrypt unregistered [ 1545.310857] Key type ._llcrypt unregistered [ 1557.861841] Key type ._llcrypt registered [ 1557.862939] Key type .llcrypt registered [ 1559.135757] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 17:30:26 (1786829426) [ 1559.547393] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1559.568400] alg: No test for adler32 (adler32-zlib) [ 1560.551456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1561.990770] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1563.753417] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 17:30:30 (1786829430) [ 1564.035542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1564.066685] LNet: 78695:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1564.085884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1564.161508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1564.181305] LNet: Added LNI 192.168.202.25@tcp [8/256/0/180] [ 1564.785335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1564.797898] LNet: Added LNI 192.168.202.25@tcp1 [8/256/0/180] [ 1565.563808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.25@tcp [ 1565.591454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.25@tcp1 [ 1565.626277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.25@tcp1 [ 1572.432661] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 17:30:39 (1786829439) [ 1574.561375] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg225-server [ 1576.925922] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 17:30:43 (1786829443) [ 1579.067762] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg225-server [ 1581.317989] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 17:30:47 (1786829447) [ 1583.375547] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg225-server [ 1585.356855] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 17:30:52 (1786829452) [ 1587.485218] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg225-server [ 1589.119794] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 17:30:56 (1786829456) [ 1590.653657] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg225-server [ 1592.163172] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 17:30:59 (1786829459) [ 1594.308742] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg225-server [ 1596.986369] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 17:31:03 (1786829463) [ 1600.589831] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg225-server [ 1603.692913] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 17:31:09 (1786829469) [ 1606.332939] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg225-server [ 1608.769869] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 17:31:15 (1786829475) [ 1611.338758] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg225-server [ 1613.920730] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 17:31:20 (1786829480) [ 1614.182383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1614.205822] LNet: 82574:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1614.213376] LNetError: 82574:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1614.221281] LNet: Removed LNI 192.168.202.25@tcp [ 1615.270101] LNet: Removed LNI 192.168.202.25@tcp1 [ 1615.286766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1615.348499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1615.399290] LNet: Added LNI 192.168.202.25@tcp [8/256/0/180] [ 1616.144739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1616.266981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.25@tcp [ 1616.519747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1616.566410] LNet: 82721:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1616.573371] LNetError: 82721:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1616.576431] LNetError: 82721:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1616.593355] LNet: Removed LNI 192.168.202.25@tcp [ 1616.624663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1616.723763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1616.772704] LNet: Added LNI 192.168.202.25@tcp [8/256/0/180] [ 1617.979838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.25@tcp [ 1618.258329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1618.300046] LNetError: 82866:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1619.368094] LNet: Removed LNI 192.168.202.25@tcp [ 1619.401292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1619.511676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1619.540082] LNet: Added LNI 192.168.202.25@tcp [8/256/0/180] [ 1620.570330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1620.617976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.25@tcp [ 1622.174651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1622.195974] LNet: 83023:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1622.204642] LNet: 83023:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1622.223254] LNetError: 83023:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1622.246637] LNet: Removed LNI 192.168.202.25@tcp [ 1622.319356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1622.475630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1622.522186] LNet: Added LNI 192.168.202.25@tcp [8/256/0/180] [ 1623.613237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.25@tcp [ 1623.937330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1625.002382] LNet: Removed LNI 192.168.202.25@tcp [ 1625.056426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1625.140396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1625.163230] LNet: Added LNI 192.168.202.25@tcp [8/256/0/180] [ 1626.006761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1626.096673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.25@tcp [ 1626.345626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1626.370118] LNet: 83316:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1626.376640] LNet: 83316:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1626.387223] LNetError: 83316:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1626.394215] LNetError: 83316:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1626.416708] LNet: Removed LNI 192.168.202.25@tcp [ 1626.473537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1626.596610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1626.658279] LNet: Added LNI 192.168.202.25@tcp [8/256/0/180] [ 1627.852186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.25@tcp [ 1628.077784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1629.161192] LNet: Removed LNI 192.168.202.25@tcp [ 1629.229368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1629.338310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1629.364819] LNet: Added LNI 192.168.202.25@tcp [8/256/0/180] [ 1630.296758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1630.364473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.25@tcp [ 1630.529720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1631.603347] LNet: Removed LNI 192.168.202.25@tcp [ 1631.687197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1631.806100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1631.836881] LNet: Added LNI 192.168.202.25@tcp [8/256/0/180] [ 1632.821836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.25@tcp [ 1633.084669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1634.153082] LNet: Removed LNI 192.168.202.25@tcp [ 1634.235143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1634.367426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1634.408922] LNet: Added LNI 192.168.202.25@tcp [8/256/0/180] [ 1635.494722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1635.620922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.25@tcp [ 1637.061169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1637.095637] LNet: 83910:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1637.106648] LNet: 83910:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 1637.113859] LNetError: 83910:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1637.120138] LNetError: 83910:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 1637.135921] LNet: Removed LNI 192.168.202.25@tcp [ 1637.166350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1637.240636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1637.272479] LNet: Added LNI 192.168.202.25@tcp [8/256/0/180] [ 1638.083742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.25@tcp [ 1638.380800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1639.464820] LNet: Removed LNI 192.168.202.25@tcp [ 1639.507374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1639.604662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1639.627901] LNet: Added LNI 192.168.202.25@tcp [8/256/0/180] [ 1640.340227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1640.422295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.25@tcp [ 1640.604661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1641.638989] LNet: Removed LNI 192.168.202.25@tcp [ 1641.664335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1641.748603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1641.783879] LNet: Added LNI 192.168.202.25@tcp [8/256/0/180] [ 1642.691735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.25@tcp [ 1644.168084] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1645.246319] LNet: Removed LNI 192.168.202.25@tcp [ 1645.284520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1645.392172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1645.416142] LNet: Added LNI 192.168.202.25@tcp [8/256/0/180] [ 1646.357155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1646.426682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1646.450772] LNet: Removed LNI 192.168.202.25@tcp [ 1646.466608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1646.539673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1646.574464] LNet: Added LNI 192.168.202.25@tcp [8/256/0/180] [ 1647.419259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1647.484266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1647.507896] LNet: Removed LNI 192.168.202.25@tcp [ 1647.520225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1647.569649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 128 [ 1647.670277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1647.707834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1647.776673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 1000000000000000000000000 [ 1647.850334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1647.885642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1647.934925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer foo [ 1648.017708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1648.046103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1648.095950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 8 [ 1648.131565] LNet: Added LNI 192.168.202.25@tcp [8/256/0/180] [ 1648.721085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1648.747276] LNet: Removed LNI 192.168.202.25@tcp [ 1648.770471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1648.840803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1656.665562] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 17:32:03 (1786829523) [ 1656.780613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1656.807102] LNet: 85372:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1656.817846] LNet: 85372:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 9 previous similar messages [ 1656.859984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1657.052766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1657.096629] LNet: Added LNI 192.168.202.25@tcp231 [8/256/0/180] [ 1658.584648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1658.618503] LNetError: 85516:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1658.623462] LNetError: 85516:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 6 previous similar messages [ 1658.648814] LNet: Removed LNI 192.168.202.25@tcp231 [ 1658.681156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1658.798311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1658.843070] LNet: Added LNI 192.168.202.25@tcp231 [8/256/0/180] [ 1658.851317] LNet: 85524:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1659.272619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1659.316377] LNet: Removed LNI 192.168.202.25@tcp231 [ 1659.347694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1659.378385] LNet: Added LNI 192.168.202.25@tcp231 [8/256/0/180] [ 1660.520432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1660.545470] LNet: Removed LNI 192.168.202.25@tcp231 [ 1660.571875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1660.659195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1660.680946] LNet: Added LNI 192.168.202.25@tcp231 [8/256/0/0] [ 1660.687019] LNet: 85677:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1660.879177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1660.901285] LNet: Removed LNI 192.168.202.25@tcp231 [ 1660.916021] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1660.940233] LNet: Added LNI 192.168.202.25@tcp231 [8/256/0/0] [ 1662.263797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1662.327141] LNet: Removed LNI 192.168.202.25@tcp231 [ 1662.372306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1662.465536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1662.488482] LNet: Added LNI 192.168.202.25@tcp231 [8/256/0/60] [ 1662.496308] LNet: 85828:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1663.065760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1663.112227] LNet: Removed LNI 192.168.202.25@tcp231 [ 1663.138900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1663.171072] LNet: Added LNI 192.168.202.25@tcp231 [8/256/0/60] [ 1664.437885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1664.470061] LNet: Removed LNI 192.168.202.25@tcp231 [ 1664.536746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1664.613885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1664.656965] LNet: Added LNI 192.168.202.25@tcp231 [8/256/0/180] [ 1664.660893] LNet: 85979:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1665.338766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1665.380093] LNet: Removed LNI 192.168.202.25@tcp231 [ 1665.416104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1665.452757] LNet: Added LNI 192.168.202.25@tcp231 [8/256/0/180] [ 1675.693483] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 17:32:22 (1786829542) [ 1675.833750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1675.896717] LNet: Removed LNI 192.168.202.25@tcp231 [ 1675.967885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1676.069318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1676.094761] LNet: Added LNI 192.168.202.25@tcp232 [8/256/0/180] [ 1676.931552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1677.047864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1677.080264] LNet: Removed LNI 192.168.202.25@tcp232 [ 1677.093745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1677.117334] LNet: Added LNI 192.168.202.25@tcp232 [8/256/0/180] [ 1678.511223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1678.588792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1678.632597] LNet: Removed LNI 192.168.202.25@tcp232 [ 1678.666804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1678.726594] LNet: Added LNI 192.168.202.25@tcp232 [8/256/0/180] [ 1680.087591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1680.169296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1680.233921] LNet: Removed LNI 192.168.202.25@tcp232 [ 1680.274334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1680.333371] LNet: Added LNI 192.168.202.25@tcp232 [8/256/0/180] [ 1682.270762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1682.417167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1682.491917] LNet: Removed LNI 192.168.202.25@tcp232 [ 1682.548796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1689.299910] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 17:32:36 (1786829556) [ 1689.640523] LNet: 87819:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1689.644322] LNet: 87819:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 5 previous similar messages [ 1690.249221] Key type .llcrypt unregistered [ 1690.256115] Key type ._llcrypt unregistered [ 1697.955481] Key type ._llcrypt registered [ 1697.958558] Key type .llcrypt registered [ 1698.186557] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1698.202637] alg: No test for adler32 (adler32-zlib) [ 1699.214268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1699.243472] LNet: Added LNI 192.168.202.25@tcp [8/256/0/180] [ 1707.890893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.125@tcp [ 1716.608427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1716.685866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1716.715665] LNet: Added LNI 192.168.202.25@tcp1 [8/256/0/180] [ 1718.263285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.25@tcp -d 192.168.202.125@tcp -m GET -e local_interrupt -r 2 [ 1718.305051] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.25@tcp -d 192.168.202.125@tcp1 -m GET -e local_interrupt -r 2 [ 1718.349608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.25@tcp -d 192.168.202.25@tcp -m GET -e local_interrupt -r 2 [ 1718.403370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.25@tcp -d 192.168.202.25@tcp1 -m GET -e local_interrupt -r 2 [ 1718.444560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.25@tcp1 -d 192.168.202.125@tcp -m GET -e local_interrupt -r 2 [ 1718.517420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.25@tcp1 -d 192.168.202.125@tcp1 -m GET -e local_interrupt -r 2 [ 1718.557440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.25@tcp1 -d 192.168.202.25@tcp -m GET -e local_interrupt -r 2 [ 1718.594358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.25@tcp1 -d 192.168.202.25@tcp1 -m GET -e local_interrupt -r 2 [ 1718.644812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.25@tcp 192.168.202.125@tcp [ 1718.673419] LNet: There was an unexpected network error while writing to 192.168.202.125: rc = -22 [ 1718.682454] LNet: 1 local NIs in recovery (showing 1): 192.168.202.25@tcp [ 1718.911445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1718.946472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1718.976437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1719.005490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1719.035308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1719.069595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1719.101967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1719.165575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1719.196068] LNet: 88485:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1719.219984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1719.243954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1719.275960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1719.308435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1719.332559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1719.396937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1719.916693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.25@tcp -d 192.168.202.125@tcp -m GET -e local_dropped -r 2 [ 1719.957983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.25@tcp -d 192.168.202.125@tcp1 -m GET -e local_dropped -r 2 [ 1720.013257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.25@tcp -d 192.168.202.25@tcp -m GET -e local_dropped -r 2 [ 1720.067047] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.25@tcp -d 192.168.202.25@tcp1 -m GET -e local_dropped -r 2 [ 1720.102338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.25@tcp1 -d 192.168.202.125@tcp -m GET -e local_dropped -r 2 [ 1720.161039] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.25@tcp1 -d 192.168.202.125@tcp1 -m GET -e local_dropped -r 2 [ 1720.213660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.25@tcp1 -d 192.168.202.25@tcp -m GET -e local_dropped -r 2 [ 1720.251041] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.25@tcp1 -d 192.168.202.25@tcp1 -m GET -e local_dropped -r 2 [ 1720.317879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.25@tcp 192.168.202.125@tcp [ 1720.336860] LNet: There was an unexpected network error while writing to 192.168.202.125: rc = -22 [ 1720.523420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1720.536657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1720.550258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1720.567880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1720.592238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1720.614836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1720.648935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1720.683708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1720.699349] LNet: 88605:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1720.713948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1720.731943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1720.758494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1720.788755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1720.833377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1720.873230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1721.447283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.25@tcp -d 192.168.202.125@tcp -m GET -e local_aborted -r 2 [ 1721.494629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.25@tcp -d 192.168.202.125@tcp1 -m GET -e local_aborted -r 2 [ 1721.534890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.25@tcp -d 192.168.202.25@tcp -m GET -e local_aborted -r 2 [ 1721.570489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.25@tcp -d 192.168.202.25@tcp1 -m GET -e local_aborted -r 2 [ 1721.604929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.25@tcp1 -d 192.168.202.125@tcp -m GET -e local_aborted -r 2 [ 1721.633625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.25@tcp1 -d 192.168.202.125@tcp1 -m GET -e local_aborted -r 2 [ 1721.680368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.25@tcp1 -d 192.168.202.25@tcp -m GET -e local_aborted -r 2 [ 1721.723647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.25@tcp1 -d 192.168.202.25@tcp1 -m GET -e local_aborted -r 2 [ 1721.770921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.25@tcp 192.168.202.125@tcp [ 1721.812942] LNet: There was an unexpected network error while writing to 192.168.202.125: rc = -22 [ 1722.142644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1722.168606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1722.195917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1722.238487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1722.277155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1722.319706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1722.362674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1722.408815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1722.428357] LNet: 88726:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1722.471490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1722.502576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1722.543049] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1722.575580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1722.615984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1722.654329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1723.254802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.25@tcp -d 192.168.202.125@tcp -m GET -e local_no_route -r 2 [ 1723.299300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.25@tcp -d 192.168.202.125@tcp1 -m GET -e local_no_route -r 2 [ 1723.325682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.25@tcp -d 192.168.202.25@tcp -m GET -e local_no_route -r 2 [ 1723.379853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.25@tcp -d 192.168.202.25@tcp1 -m GET -e local_no_route -r 2 [ 1723.440307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.25@tcp1 -d 192.168.202.125@tcp -m GET -e local_no_route -r 2 [ 1723.482658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.25@tcp1 -d 192.168.202.125@tcp1 -m GET -e local_no_route -r 2 [ 1723.534437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.25@tcp1 -d 192.168.202.25@tcp -m GET -e local_no_route -r 2 [ 1723.582448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.25@tcp1 -d 192.168.202.25@tcp1 -m GET -e local_no_route -r 2 [ 1723.623220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.25@tcp 192.168.202.125@tcp [ 1723.910976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1723.964272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1724.016215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1724.046771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1724.069965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1724.093206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1724.140328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1724.174331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1724.215953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1724.252435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1724.304433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1724.351979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1724.418311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1724.471538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1725.072910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.25@tcp -d 192.168.202.125@tcp -m GET -e local_timeout -r 2 [ 1725.110078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.25@tcp -d 192.168.202.125@tcp1 -m GET -e local_timeout -r 2 [ 1725.155992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.25@tcp -d 192.168.202.25@tcp -m GET -e local_timeout -r 2 [ 1725.190405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.25@tcp -d 192.168.202.25@tcp1 -m GET -e local_timeout -r 2 [ 1725.240411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.25@tcp1 -d 192.168.202.125@tcp -m GET -e local_timeout -r 2 [ 1725.265293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.25@tcp1 -d 192.168.202.125@tcp1 -m GET -e local_timeout -r 2 [ 1725.293833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.25@tcp1 -d 192.168.202.25@tcp -m GET -e local_timeout -r 2 [ 1725.321638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.25@tcp1 -d 192.168.202.25@tcp1 -m GET -e local_timeout -r 2 [ 1725.353896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.25@tcp 192.168.202.125@tcp [ 1725.372157] LNet: There was an unexpected network error while writing to 192.168.202.125: rc = -22 [ 1725.375828] LNet: Skipped 1 previous similar message [ 1725.620934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1725.661712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1725.686781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1725.704441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1725.722885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1725.797993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1725.837565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1725.878280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1725.902669] LNet: 88966:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1725.917097] LNet: 88966:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1725.943989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1725.986746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1726.019974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1726.046216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1726.091927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1726.139970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1726.814730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.25@tcp -d 192.168.202.125@tcp -m GET -e remote_dropped -r 2 [ 1726.855743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.25@tcp -d 192.168.202.125@tcp1 -m GET -e remote_dropped -r 2 [ 1726.927556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.25@tcp -d 192.168.202.25@tcp -m GET -e remote_dropped -r 2 [ 1726.998920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.25@tcp -d 192.168.202.25@tcp1 -m GET -e remote_dropped -r 2 [ 1727.028972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.25@tcp1 -d 192.168.202.125@tcp -m GET -e remote_dropped -r 2 [ 1727.102332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.25@tcp1 -d 192.168.202.125@tcp1 -m GET -e remote_dropped -r 2 [ 1727.176160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.25@tcp1 -d 192.168.202.25@tcp -m GET -e remote_dropped -r 2 [ 1727.228915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.25@tcp1 -d 192.168.202.25@tcp1 -m GET -e remote_dropped -r 2 [ 1727.275614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.25@tcp 192.168.202.125@tcp [ 1727.328191] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.125@tcp [ 1727.601904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1727.622918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1727.657315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1727.687323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1727.717335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1727.747139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1727.798375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1727.827417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1727.860160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1727.895212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1727.923216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1727.954379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1727.985949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1728.013520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1734.258768] LNet: 89164:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1734.264382] LNetError: 89164:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1734.273778] LNet: Removed LNI 192.168.202.25@tcp [ 1734.307354] LNet: Removed LNI 192.168.202.25@tcp1 [ 1734.670562] Key type .llcrypt unregistered [ 1734.672064] Key type ._llcrypt unregistered [ 1750.037957] Key type ._llcrypt registered [ 1750.041821] Key type .llcrypt registered [ 1751.218853] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 17:33:38 (1786829618) [ 1751.510419] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1751.523946] alg: No test for adler32 (adler32-zlib) [ 1752.520123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1752.564986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1752.602056] LNet: Added LNI 192.168.202.25@tcp [8/256/0/180] [ 1753.545411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1753.575431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.202.25@tcp [ 1753.604422] LNetError: 90142:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.202.25@tcp to recovery [ 1753.702749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.202.25@tcp [ 1753.819210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1760.887568] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 17:33:48 (1786829628) [ 1761.165939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1761.188817] LNet: 90732:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1761.197873] LNetError: 90732:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1761.213688] LNet: Removed LNI 192.168.202.25@tcp [ 1761.225911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1761.274362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1761.302635] LNet: Added LNI 192.168.202.25@tcp [8/256/0/180] [ 1761.902756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1761.930229] LNet: Added LNI 192.168.202.25@tcp1 [8/256/0/180] [ 1762.737940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.202.25@tcp [ 1762.760563] LNetError: 90999:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.202.25@tcp to recovery [ 1763.297778] LNet: 1 local NIs in recovery (showing 1): 192.168.202.25@tcp [ 1763.304768] LNet: There was an unexpected network error while writing to 192.168.202.25: rc = -22 [ 1764.052822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.202.25@tcp [ 1773.594866] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 17:34:00 (1786829640) [ 1776.171365] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg225-server [ 1778.738767] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 17:34:05 (1786829645) [ 1782.500570] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg225-server [ 1785.412019] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 17:34:11 (1786829651) [ 1788.861506] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg225-server [ 1791.715614] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 17:34:18 (1786829658) [ 1791.820086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1791.894719] LNet: 92625:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1791.913677] LNetError: 92625:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1791.924830] LNet: Removed LNI 192.168.202.25@tcp [ 1791.943398] LNet: Removed LNI 192.168.202.25@tcp1 [ 1791.999683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1792.195974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1792.269059] LNet: Added LNI 192.168.202.25@tcp [8/256/0/180] [ 1793.545773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1793.658044] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1793.771154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1793.989498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1794.120974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1794.192199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1802.116596] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 17:34:29 (1786829669) [ 1802.236965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1802.271345] LNet: 93349:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1802.279710] LNetError: 93349:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1802.288331] LNetError: 93349:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1802.310950] LNet: Removed LNI 192.168.202.25@tcp [ 1802.341588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1802.412466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1802.430961] LNet: Added LNI 192.168.202.25@tcp [8/256/0/180] [ 1809.376674] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 17:34:36 (1786829676) [ 1809.814198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1809.828730] LNet: 94061:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1809.834739] LNetError: 94061:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1809.843375] LNet: Removed LNI 192.168.202.25@tcp [ 1809.859679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1809.898233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1809.920386] LNet: Added LNI 192.168.202.25@tcp [8/256/0/180] [ 1817.362745] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 17:34:44 (1786829684) [ 1818.930341] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 1820.519498] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 17:34:47 (1786829687) [ 1821.131971] LNet: 95095:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1821.135760] LNetError: 95095:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1821.144887] LNet: Removed LNI 192.168.202.25@tcp [ 1821.809976] Key type .llcrypt unregistered [ 1821.814721] Key type ._llcrypt unregistered [ 1828.299707] Key type ._llcrypt registered [ 1828.301325] Key type .llcrypt registered [ 1828.643281] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1828.667272] alg: No test for adler32 (adler32-zlib) [ 1829.590563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1829.624560] LNet: Added LNI 192.168.202.25@tcp [8/256/0/180] [ 1836.791364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.125@tcp [ 1842.536857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.202.125@tcp [ 1843.135327] LNet: 95520:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1843.140953] LNetError: 95520:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1843.164208] LNet: Removed LNI 192.168.202.25@tcp [ 1843.607587] Key type .llcrypt unregistered [ 1843.610105] Key type ._llcrypt unregistered [ 1854.270778] Key type ._llcrypt registered [ 1854.276132] Key type .llcrypt registered [ 1855.331536] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 1856.737384] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 1858.328925] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 17:35:25 (1786829725) [ 1860.122466] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 [ 1861.643416] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 17:35:28 (1786829728) [ 1863.485581] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg225-server [ 1864.949887] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 17:35:32 (1786829732) [ 1866.420409] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg225-server [ 1867.650265] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 17:35:35 (1786829735) [ 1869.146390] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1869.171653] alg: No test for adler32 (adler32-zlib) [ 1870.213166] Lustre: Lustre: Build Version: 2.17.57_3_g928f38d [ 1870.416987] LNet: Added LNI 192.168.202.25@tcp [8/256/0/180] [ 1872.042059] Key type lgssc registered [ 1873.153772] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1894.421937] Key type lgssc unregistered [ 1894.754738] LNet: 98697:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1894.764753] LNetError: 98697:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1894.788354] LNet: Removed LNI 192.168.202.25@tcp [ 1895.414138] Key type .llcrypt unregistered [ 1895.418708] Key type ._llcrypt unregistered [ 1913.650317] Key type ._llcrypt registered [ 1913.651692] Key type .llcrypt registered [ 1914.597288] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 17:36:21 (1786829781) [ 1915.146322] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1915.165675] alg: No test for adler32 (adler32-zlib) [ 1916.070990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1916.109535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1916.140975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1916.184525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1916.230920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1922.605914] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 17:36:29 (1786829789) [ 1922.722783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1922.759223] LNet: 100106:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1922.791287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1922.882679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1922.902513] LNet: Added LNI 192.168.202.25@tcp [8/256/0/180] [ 1923.891378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1923.917078] LNet: 100256:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1923.927435] LNetError: 100256:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1923.969239] LNet: Removed LNI 192.168.202.25@tcp [ 1924.027194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1924.178391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1924.251456] LNet: Added LNI 192.168.202.25@tcp [8/256/0/180] [ 1925.200803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1925.231457] LNet: 100284:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1925.248092] LNetError: 100284:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1925.261766] LNet: Removed LNI 192.168.202.25@tcp [ 1925.293616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1925.370397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1925.437667] LNet: Added LNI 192.168.202.25@tcp [8/256/0/180] [ 1933.010851] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 17:36:40 (1786829800) [ 1933.321812] LNet: 100875:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1933.329641] LNetError: 100875:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1933.356671] LNet: Removed LNI 192.168.202.25@tcp [ 1934.124733] Key type .llcrypt unregistered [ 1934.128297] Key type ._llcrypt unregistered [ 1940.796351] Key type ._llcrypt registered [ 1940.797978] Key type .llcrypt registered [ 1941.109625] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1941.147762] alg: No test for adler32 (adler32-zlib) [ 1942.157421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1942.275472] LNet: Added LNI 192.168.202.25@tcp [8/256/0/180] [ 1942.278920] LNet: *** cfs_fail_loc=e003, val=100*** [ 1943.009428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.25@tcp [ 1943.032573] LNet: *** cfs_fail_loc=e003, val=100*** [ 1943.036113] LNet: Skipped 2 previous similar messages [ 1943.042443] Ping reply received with mapping: [ 1943.045417] found metadata reply NI[192.168.202.25@tcp] [ 1943.049081] 19 ca a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 1943.052202] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 1943.054743] 5a 5a 5a 5a 5a 5a 5a 5a [ 1943.226515] LNet: 101276:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1943.235304] LNetError: 101276:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1944.304104] LNet: Removed LNI 192.168.202.25@tcp [ 1945.248587] Key type .llcrypt unregistered [ 1945.254994] Key type ._llcrypt unregistered [ 1957.590327] Key type ._llcrypt registered [ 1957.593201] Key type .llcrypt registered [ 1959.184267] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 17:37:05 (1786829825) [ 1960.181302] Key type .llcrypt unregistered [ 1960.183074] Key type ._llcrypt unregistered [ 1968.812451] Key type ._llcrypt registered [ 1968.819568] Key type .llcrypt registered [ 1968.954279] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1968.969487] alg: No test for adler32 (adler32-zlib) [ 1970.260513] LNetError: 102336:0:(api-ni.c:2850:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 1971.048996] Key type .llcrypt unregistered [ 1971.050831] Key type ._llcrypt unregistered [ 1978.023873] Key type ._llcrypt registered [ 1978.025772] Key type .llcrypt registered [ 1979.847340] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 17:37:26 (1786829846) [ 1980.444438] Key type .llcrypt unregistered [ 1980.445988] Key type ._llcrypt unregistered [ 1987.062137] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1987.079888] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1989.938297] Key type ._llcrypt registered [ 1989.941469] Key type .llcrypt registered [ 1990.264480] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1990.290681] alg: No test for adler32 (adler32-zlib) [ 1991.327724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1991.382736] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2001.282421] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 17:37:48 (1786829868) [ 2001.404913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2001.433644] LNet: 104122:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2001.441778] LNetError: 104122:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2001.465257] LNet: Removed LNI 10.1.2.3@tcp [ 2001.503743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2002.462468] Key type .llcrypt unregistered [ 2002.465697] Key type ._llcrypt unregistered [ 2009.937272] Key type ._llcrypt registered [ 2009.938814] Key type .llcrypt registered [ 2010.151850] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2010.165376] alg: No test for adler32 (adler32-zlib) [ 2011.091767] LNet: Added LNI 192.168.202.25@tcp [8/256/0/180] [ 2011.159664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.25@tcp [ 2011.367504] LNet: 104419:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2011.379317] LNetError: 104419:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2011.395652] LNet: Removed LNI 192.168.202.25@tcp [ 2012.113424] Key type .llcrypt unregistered [ 2012.115995] Key type ._llcrypt unregistered [ 2023.220785] Key type ._llcrypt registered [ 2023.225295] Key type .llcrypt registered [ 2025.006990] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 17:38:11 (1786829891) [ 2025.670001] Key type .llcrypt unregistered [ 2025.675144] Key type ._llcrypt unregistered [ 2033.511251] Key type ._llcrypt registered [ 2033.514882] Key type .llcrypt registered [ 2033.741600] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2033.749930] alg: No test for adler32 (adler32-zlib) [ 2041.238685] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 17:38:28 (1786829908) [ 2041.412802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2041.496887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2041.560513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2041.587094] LNet: Added LNI 192.168.202.25@tcp [8/256/0/180] [ 2042.487668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2042.517669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 2042.559289] LNet: Removed LNI 192.168.202.25@tcp [ 2042.894806] LNet: 106176:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2043.322917] Key type .llcrypt unregistered [ 2043.324444] Key type ._llcrypt unregistered [ 2055.402246] Key type ._llcrypt registered [ 2055.406934] Key type .llcrypt registered [ 2056.625446] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 17:38:43 (1786829923) [ 2057.155919] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2057.182195] alg: No test for adler32 (adler32-zlib) [ 2058.097659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2058.138687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2058.158691] LNet: Added LNI 192.168.202.25@tcp [8/256/0/180] [ 2058.731903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.25@tcp [ 2058.761806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.202.25@tcp [ 2058.777178] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 2058.782384] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 2058.785618] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2058, 0, 0 [ 2058.790362] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 2058.792631] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 2058.795529] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 2058.797966] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2058, 0, 0 [ 2058.802160] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 2058.803990] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2058.807247] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 2058.810347] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2058, 0, 2108 [ 2058.814188] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 2058.816461] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2058.819624] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 2058.822251] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2058, 0, 2108 [ 2058.826120] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 2064.887393] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 17:38:52 (1786829932) [ 2065.465991] LNet: 107714:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2065.471841] LNetError: 107714:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2066.535873] LNet: Removed LNI 192.168.202.25@tcp [ 2067.351441] Key type .llcrypt unregistered [ 2067.355649] Key type ._llcrypt unregistered [ 2074.681384] Key type ._llcrypt registered [ 2074.684822] Key type .llcrypt registered [ 2074.868476] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2074.890253] alg: No test for adler32 (adler32-zlib) [ 2075.836979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2075.858632] LNet: Added LNI 192.168.202.25@tcp [8/256/0/180] [ 2085.360811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.125@tcp [ 2096.549849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2096.628638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2096.674264] LNet: Added LNI 192.168.202.25@tcp1 [8/256/0/180] [ 2098.151419] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2098.331323] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2098.354622] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2098.368648] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2101.217984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 2101.274835] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 2112.866593] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 17:39:39 (1786829979) [ 2113.058843] LNet: 109229:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2113.063784] LNetError: 109229:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2113.071980] LNet: Removed LNI 192.168.202.25@tcp [ 2114.080403] LNet: Removed LNI 192.168.202.25@tcp1 [ 2114.085632] LNetError: 109229:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2114.092037] LNetError: 109229:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2114.108378] LNet: Removed LNI 10.1.2.3@tcp99 [ 2114.815291] Key type .llcrypt unregistered [ 2114.821301] Key type ._llcrypt unregistered [ 2121.221888] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2121.225824] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2123.727729] Key type ._llcrypt registered [ 2123.729443] Key type .llcrypt registered [ 2123.874642] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2123.883549] alg: No test for adler32 (adler32-zlib) [ 2124.902377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2124.989345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2125.006994] LNet: Added LNI 192.168.202.25@tcp [8/256/0/180] [ 2125.592952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2125.619760] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2127.682670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 2127.721665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2127.763927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 2127.795966] LNetError: 109978:0:(peer.c:2362:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 2127.836826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2127.894608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.25@tcp [ 2128.315932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2128.403130] LNet: 109998:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2128.412217] LNetError: 109998:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2128.416870] LNet: Removed LNI 10.1.2.3@tcp [ 2129.460240] LNet: Removed LNI 192.168.202.25@tcp [ 2130.428781] Key type .llcrypt unregistered [ 2130.433567] Key type ._llcrypt unregistered [ 2147.226802] Key type ._llcrypt registered [ 2147.230543] Key type .llcrypt registered [ 2149.207073] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 17:40:15 (1786830015) [ 2149.626780] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2149.656664] alg: No test for adler32 (adler32-zlib) [ 2150.624589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2150.687689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2150.715120] LNet: Added LNI 192.168.202.25@tcp [8/256/0/180] [ 2151.890887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.25@tcp [ 2152.073555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg225-client@tcp [ 2159.652284] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 17:40:26 (1786830026) [ 2159.961732] LNet: 111537:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2159.973719] LNetError: 111537:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2160.012230] LNet: Removed LNI 192.168.202.25@tcp [ 2160.597058] Key type .llcrypt unregistered [ 2160.600247] Key type ._llcrypt unregistered [ 2168.165602] Key type ._llcrypt registered [ 2168.167251] Key type .llcrypt registered [ 2168.490683] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2168.522405] alg: No test for adler32 (adler32-zlib) [ 2169.513979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 2169.550353] LNet: Added LNI 192.168.202.25@tcp [8/256/0/180] [ 2169.589181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 2177.259505] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 17:40:44 (1786830044) [ 2179.201632] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 2181.141278] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 17:40:48 (1786830048) [ 2182.974602] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 2185.009201] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 17:40:51 (1786830051) [ 2185.226161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2185.265538] LNet: 112960:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2185.275483] LNetError: 112960:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2185.285601] LNet: Removed LNI 192.168.202.25@tcp [ 2185.313197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2185.405471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2185.425556] LNet: Added LNI 192.168.202.25@tcp [8/256/0/180] [ 2186.195568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2186.223548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2193.244026] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 17:41:00 (1786830060) [ 2193.430743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2193.458918] LNet: 113669:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2193.466946] LNetError: 113669:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2193.502121] LNet: Removed LNI 192.168.202.25@tcp [ 2193.533741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2193.584468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 2193.639694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2194.485568] Key type .llcrypt unregistered [ 2194.490089] Key type ._llcrypt unregistered [ 2207.430678] Key type ._llcrypt registered [ 2207.432767] Key type .llcrypt registered [ 2208.551430] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 17:41:15 (1786830075) [ 2208.942669] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2208.955399] alg: No test for adler32 (adler32-zlib) [ 2209.823594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2209.873150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2209.903196] LNet: Added LNI 192.168.202.25@tcp [8/256/0/180] [ 2210.608814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2210.659978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.25@tcp [ 2217.632527] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 17:41:24 (1786830084) [ 2217.752520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2217.793448] LNet: 115223:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2217.807431] LNetError: 115223:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2217.830682] LNet: Removed LNI 192.168.202.25@tcp [ 2217.862601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2217.944582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 2217.967993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 2225.227512] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 17:41:32 (1786830092) [ 2225.321153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2225.363362] LNet: 115802:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2225.398705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2225.514141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2225.876793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2225.902886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2225.964767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 2226.361671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2226.419705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2226.511215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2226.532624] LNet: 115842:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2226.558490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2226.636989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2227.359155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2227.419585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2227.504685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 2228.249770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2228.321855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2235.550867] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 17:41:42 (1786830102) [ 2235.747914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2235.796721] LNet: 116441:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2235.815739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2235.856533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2235.872596] LNet: Added LNI 192.168.202.25@tcp [8/256/0/180] [ 2236.483595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2236.513806] LNet: Added LNI 192.168.202.25@tcp2 [8/256/0/180] [ 2237.700832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2237.831665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.25@tcp [ 2238.097108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2238.125072] LNetError: 116722:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2238.132617] LNet: Removed LNI 192.168.202.25@tcp [ 2238.151129] LNet: Removed LNI 192.168.202.25@tcp2 [ 2238.178692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2238.276493] LNet: Added LNI 192.168.202.25@tcp [8/256/0/180] [ 2238.285883] LNet: Added LNI 192.168.202.25@tcp2 [8/256/0/180] [ 2238.301437] LNet: 116728:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2239.126648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.25@tcp [ 2239.465814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2239.493940] LNetError: 116765:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2239.500130] LNetError: 116765:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2239.518046] LNet: Removed LNI 192.168.202.25@tcp [ 2239.536658] LNet: Removed LNI 192.168.202.25@tcp2 [ 2239.617128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2239.752961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2239.793222] LNet: Added LNI 192.168.202.25@tcp [8/256/0/180] [ 2240.789669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2240.827363] LNet: Added LNI 192.168.202.25@tcp2 [8/256/0/180] [ 2241.904743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.25@tcp [ 2241.944912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2242.002671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.25@tcp [ 2242.324950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2242.374969] LNet: 117050:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2242.390994] LNet: 117050:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 2242.405744] LNetError: 117050:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2242.418389] LNetError: 117050:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2242.433737] LNet: Removed LNI 192.168.202.25@tcp [ 2243.507378] LNet: Removed LNI 192.168.202.25@tcp2 [ 2243.546342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2243.635796] LNet: Added LNI 192.168.202.25@tcp [8/256/0/180] [ 2243.649638] LNet: Added LNI 192.168.202.25@tcp2 [8/256/0/180] [ 2243.658050] LNet: 117056:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2244.523396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.25@tcp [ 2244.758635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2244.780349] LNet: Removed LNI 192.168.202.25@tcp [ 2245.815582] LNet: Removed LNI 192.168.202.25@tcp2 [ 2245.849066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2245.913659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2245.956474] LNet: Added LNI 192.168.202.25@tcp [8/256/0/180] [ 2246.660082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2246.677924] LNet: Added LNI 192.168.202.25@tcp2 [8/256/0/180] [ 2247.771471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2247.857875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.25@tcp [ 2248.222770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2248.263367] LNetError: 117373:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2248.269271] LNetError: 117373:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 2249.314067] LNet: Removed LNI 192.168.202.25@tcp [ 2249.345894] LNet: Removed LNI 192.168.202.25@tcp2 [ 2249.424052] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2249.605659] LNet: Added LNI 192.168.202.25@tcp [8/256/0/180] [ 2249.617462] LNet: Added LNI 192.168.202.25@tcp2 [8/256/0/180] [ 2249.627762] LNet: 117379:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2250.686695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.25@tcp [ 2251.027093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2251.065634] LNet: 117416:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2251.072000] LNet: 117416:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 2252.128305] LNet: Removed LNI 192.168.202.25@tcp [ 2252.138111] LNet: Removed LNI 192.168.202.25@tcp2 [ 2252.192022] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2252.297359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2252.336682] LNet: Added LNI 192.168.202.25@tcp [8/256/0/180] [ 2253.545077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2253.570198] LNet: Added LNI 192.168.202.25@tcp2 [8/256/0/180] [ 2254.601547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.25@tcp [ 2254.657083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2254.739846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.25@tcp [ 2255.133180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2256.224717] LNet: Removed LNI 192.168.202.25@tcp [ 2257.257364] LNet: Removed LNI 192.168.202.25@tcp2 [ 2257.283955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2257.348471] LNet: Added LNI 192.168.202.25@tcp [8/256/0/180] [ 2257.353922] LNet: Added LNI 192.168.202.25@tcp2 [8/256/0/180] [ 2257.359966] LNet: 117707:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2257.982496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.25@tcp [ 2258.261956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2258.293564] LNetError: 117744:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2258.298443] LNetError: 117744:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 2258.302869] LNet: Removed LNI 192.168.202.25@tcp [ 2259.367494] LNet: Removed LNI 192.168.202.25@tcp2 [ 2259.420533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2259.512291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2259.543830] LNet: Added LNI 192.168.202.25@tcp [8/256/0/180] [ 2260.434721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2260.452396] LNet: Added LNI 192.168.202.25@tcp2 [8/256/0/180] [ 2261.385355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2261.426483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2261.491944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.25@tcp [ 2261.910726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2262.944563] LNet: Removed LNI 192.168.202.25@tcp [ 2262.969385] LNet: Removed LNI 192.168.202.25@tcp2 [ 2263.023219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2263.138211] LNet: Added LNI 192.168.202.25@tcp [8/256/0/180] [ 2263.147912] LNet: Added LNI 192.168.202.25@tcp2 [8/256/0/180] [ 2263.165402] LNet: 118047:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2263.988864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.25@tcp [ 2264.352886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2265.440377] LNet: Removed LNI 192.168.202.25@tcp [ 2265.449504] LNet: Removed LNI 192.168.202.25@tcp2 [ 2265.492393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2265.695597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2265.738650] LNet: Added LNI 192.168.202.25@tcp [8/256/0/180] [ 2266.830504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2266.855695] LNet: Added LNI 192.168.202.25@tcp2 [8/256/0/180] [ 2268.224811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.25@tcp [ 2268.325093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2268.370327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2268.461225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.25@tcp [ 2269.001853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2269.022496] LNet: 118400:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2269.028453] LNet: 118400:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 2270.050384] LNet: Removed LNI 192.168.202.25@tcp [ 2271.151451] LNet: Removed LNI 192.168.202.25@tcp2 [ 2271.233821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2271.326134] LNet: Added LNI 192.168.202.25@tcp [8/256/0/180] [ 2271.348180] LNet: Added LNI 192.168.202.25@tcp2 [8/256/0/180] [ 2271.356899] LNet: 118407:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2272.580805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.25@tcp [ 2281.205835] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 2283.778794] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 17:42:30 (1786830150) [ 2283.936846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2283.992356] LNetError: 119160:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2284.011914] LNetError: 119160:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 2284.032150] LNet: Removed LNI 192.168.202.25@tcp [ 2285.095402] LNet: Removed LNI 192.168.202.25@tcp2 [ 2285.181711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2285.694856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2285.762556] LNet: Added LNI 192.168.202.25@tcp [8/256/0/180] [ 2286.630853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2286.690698] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2287.884535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2287.912549] LNet: Removed LNI 10.1.2.3@tcp [ 2287.932498] LNet: Removed LNI 192.168.202.25@tcp [ 2287.963537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2288.082527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 2288.112622] LNet: Added LNI 192.168.202.25@tcp [8/256/0/180] [ 2288.116511] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2288.633491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 2288.670589] LNet: Removed LNI 192.168.202.25@tcp [ 2288.698720] LNet: Removed LNI 10.1.2.3@tcp [ 2295.132636] Lustre: DEBUG MARKER: == sanity-lnet test 475: per-operation latency stats populate and reset ========================================================== 17:42:42 (1786830162) [ 2295.552131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2295.637959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2295.720987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2295.741627] LNet: Added LNI 192.168.202.25@tcp [8/256/0/180] [ 2296.872787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2296.952966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.25@tcp [ 2297.015138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.25@tcp [ 2297.078801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.25@tcp [ 2297.140468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.25@tcp [ 2297.196531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.25@tcp [ 2297.240564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.25@tcp [ 2297.299690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.25@tcp [ 2297.379568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.25@tcp [ 2297.441332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.25@tcp [ 2297.505448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.25@tcp [ 2297.802856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2305.213533] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 17:42:52 (1786830172) [ 2305.536356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2305.595844] LNet: 120781:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2305.610931] LNet: 120781:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 2306.664375] LNet: Removed LNI 192.168.202.25@tcp [ 2306.713532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2306.846713] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2306.977532] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2306.987494] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2307.001170] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2309.554192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2309.590897] LNet: Added LNI 192.168.202.25@tcp [8/256/0/180] [ 2310.626324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2310.671519] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2313.259045] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.25@tcp [ 2313.314213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 2313.445390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.202.25@tcp -m PUT -e local_timeout -r 1 [ 2313.517970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 2313.601653] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2313.607240] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2313.923115] LNetError: 120845:0:(lib-move.c:4801:lnet_parse()) 192.168.202.25@tcp, src 192.168.202.25@tcp: Dropping PUT (error -108 looking up sender) [ 2314.913170] LNet: Removed LNI 10.1.2.3@tcp [ 2315.942523] LNet: Removed LNI 192.168.202.25@tcp [ 2316.344384] Key type .llcrypt unregistered [ 2316.346708] Key type ._llcrypt unregistered [ 2331.382614] Key type ._llcrypt registered [ 2331.387291] Key type .llcrypt registered [ 2333.388960] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 17:43:19 (1786830199) [ 2334.179848] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2334.223594] alg: No test for adler32 (adler32-zlib) [ 2335.313638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2336.069492] LNet: 122161:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2337.094644] Key type .llcrypt unregistered [ 2337.098298] Key type ._llcrypt unregistered [ 2347.568932] Key type ._llcrypt registered [ 2347.573742] Key type .llcrypt registered [ 2347.725797] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2347.750145] alg: No test for adler32 (adler32-zlib) [ 2348.757178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2348.809439] LNet: Added LNI 192.168.202.25@tcp [8/256/0/180] [ 2357.759472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.125@tcp [ 2357.994290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 2358.692955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.202.125@tcp [ 2358.998248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2359.055864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2359.075847] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.125@tcp [ 2359.101915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2359.180113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2359.246735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2359.307438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2359.376663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2359.412674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2359.431788] LNet: 122616:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2359.455097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2359.495266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2359.529284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2359.557542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2359.575486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2359.600851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2360.331576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2360.583617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2360.625285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2360.647804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2360.668411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2360.705520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2360.745415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2360.781554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2360.818384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2360.837808] LNet: 122733:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2360.865287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2360.898868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2360.934977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2360.976606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2361.005134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2361.027925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2361.693892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.202.125@tcp [ 2362.063850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2362.119070] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2362.163863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2362.204562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2362.228296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2362.266552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2362.316325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2362.359982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2362.388841] LNet: 122850:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2362.419416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2362.454432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2362.486506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2362.515671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2362.553551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2362.582585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2363.344446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2363.673945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2363.722276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2363.771610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2363.809790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2363.849567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2363.900497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2363.952375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2363.981378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2364.022785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2364.091594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2364.130672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2364.168867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2364.215398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2364.251963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2364.723993] LNet: 123017:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2364.731348] LNetError: 123017:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2364.751663] LNet: Removed LNI 192.168.202.25@tcp [ 2365.264920] Key type .llcrypt unregistered [ 2365.266783] Key type ._llcrypt unregistered [ 2386.273521] Key type ._llcrypt registered [ 2386.278291] Key type .llcrypt registered [ 2387.901838] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 17:44:14 (1786830254) [ 2388.454492] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2388.468533] alg: No test for adler32 (adler32-zlib) [ 2389.464655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2389.853673] LNet: 123905:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2390.491338] Key type .llcrypt unregistered [ 2390.493250] Key type ._llcrypt unregistered [ 2397.603665] Key type ._llcrypt registered [ 2397.605823] Key type .llcrypt registered [ 2397.994284] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2398.017354] alg: No test for adler32 (adler32-zlib) [ 2399.013320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2399.052131] LNet: Added LNI 192.168.202.25@tcp [8/256/0/180] [ 2407.738262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.125@tcp [ 2417.640670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2417.706862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2417.737163] LNet: Added LNI 192.168.202.25@tcp1 [8/256/0/180] [ 2419.253091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 2419.993584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.202.125@tcp [ 2420.320515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2420.387310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2420.437429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2420.488268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2420.534817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2420.557830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2420.584101] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.125@tcp [ 2420.591961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2420.623544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2420.652891] LNet: 124561:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2420.684418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2420.744499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2420.796542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2420.863723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2420.927529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2420.982493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2422.040513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2422.278417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2422.310895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2422.347668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2422.379234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2422.429837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2422.467871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2422.507464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2422.539526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2422.566150] LNet: 124679:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2422.597714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2422.632182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2422.663405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2422.695416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2422.722815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2422.763912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2423.566342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.202.125@tcp [ 2423.885529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2423.916918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2423.949678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2423.986432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2424.023497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2424.071867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2424.124417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2424.214391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2424.249516] LNet: 124796:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2424.280129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2424.323241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2424.388899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2424.415968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2424.487333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2424.538205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2425.333041] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2425.608916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2425.648226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2425.690322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2425.733180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2425.792875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2425.834491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2425.890556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2425.925762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2425.965401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2426.020928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2426.069186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2426.111454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2426.177773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2426.249634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2431.750190] LNet: 124998:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2431.760949] LNetError: 124998:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2431.767054] LNet: Removed LNI 192.168.202.25@tcp [ 2431.781782] LNet: Removed LNI 192.168.202.25@tcp1 [ 2432.255928] Key type .llcrypt unregistered [ 2432.258275] Key type ._llcrypt unregistered [ 2451.535236] Key type ._llcrypt registered [ 2451.539992] Key type .llcrypt registered [ 2452.912828] Lustre: DEBUG MARKER: == sanity-lnet test 525: keep reachable peer on local net delete ========================================================== 17:45:19 (1786830319) [ 2453.282360] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2453.305792] alg: No test for adler32 (adler32-zlib) [ 2454.294821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2454.382838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2454.412207] LNet: Added LNI 192.168.202.25@tcp [8/256/0/180] [ 2455.182752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2455.211804] LNet: Added LNI 192.168.202.25@tcp2 [8/256/0/180] [ 2455.960670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.25@tcp [ 2456.069887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp [ 2457.120283] LNet: Removed LNI 192.168.202.25@tcp [ 2465.182082] Lustre: DEBUG MARKER: == sanity-lnet test 550: DD-off: keep route up when gw reachable via another local NI ========================================================== 17:45:31 (1786830331) [ 2467.004322] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_550 Need at least 2 local interfaces for multi-rail [ 2469.344779] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 17:45:35 (1786830335) [ 2477.269750] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 17:45:44 (1786830344) [ 2486.169288] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 17:45:52 (1786830352) [ 2495.691438] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 17:46:02 (1786830362) [ 2504.583791] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 17:46:11 (1786830371) [ 2514.769712] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 17:46:21 (1786830381) [ 2523.832835] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 17:46:30 (1786830390) [ 2534.970788] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 17:46:40 (1786830400) [ 2545.224697] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 17:46:51 (1786830411) [ 2554.538834] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 17:47:01 (1786830421) [ 2565.998061] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 17:47:11 (1786830431) [ 2574.934624] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 17:47:21 (1786830441) [ 2583.983710] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 17:47:30 (1786830450) [ 2594.481944] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 17:47:41 (1786830461) [ 2603.107242] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 17:47:50 (1786830470) [ 2611.897627] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 17:47:58 (1786830478) [ 2620.309638] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 17:48:07 (1786830487) [ 2629.857881] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 17:48:16 (1786830496) [ 2630.141684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2630.272478] LNet: 136693:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2630.299821] LNetError: 136693:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2630.344667] LNet: Removed LNI 192.168.202.25@tcp2 [ 2630.450532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2630.627717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2630.687350] LNet: Added LNI 192.168.202.25@tcp [8/256/0/180] [ 2632.464802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2632.491382] LNet: 136840:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2632.505802] LNetError: 136840:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2632.527297] LNet: Removed LNI 192.168.202.25@tcp [ 2632.550180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2632.625615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2632.687921] LNet: Added LNI 192.168.202.25@tcp [8/256/0/180] [ 2640.969555] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 17:48:27 (1786830507) [ 2650.796960] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 17:48:37 (1786830517) [ 2659.987377] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 17:48:46 (1786830526) [ 2668.614877] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 17:48:55 (1786830535) [ 2678.280277] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 17:49:05 (1786830545) [ 2687.033827] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 17:49:13 (1786830553) [ 2695.157503] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 17:49:21 (1786830561) [ 2705.215302] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 17:49:31 (1786830571) [ 2717.797880] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 17:49:44 (1786830584) [ 2726.177532] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 17:49:52 (1786830592) [ 2735.174983] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 17:50:01 (1786830601) [ 2742.791541] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 17:50:10 (1786830610) [ 2751.374962] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 2542 sec ========== 17:50:18 (1786830618) [ 2751.783964] LNet: 144310:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2751.794738] LNetError: 144310:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2751.808583] LNet: Removed LNI 192.168.202.25@tcp [ 2752.618864] Key type .llcrypt unregistered [ 2752.620521] Key type ._llcrypt unregistered [ 2766.533385] Key type ._llcrypt registered [ 2766.554435] Key type .llcrypt registered [ 2767.044073] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2767.067768] alg: No test for adler32 (adler32-zlib) [ 2768.342407] Lustre: Lustre: Build Version: 2.17.57_3_g928f38d [ 2768.701380] LNet: Added LNI 192.168.202.25@tcp [8/256/0/180] [ 2770.360200] Key type lgssc registered [ 2771.795551] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2849.735599] Lustre: Mounted lustre-client - version 2.17.57_3_g928f38d [ 2855.540920] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2864.554242] Lustre: Unmounted lustre-client [ 2889.442312] Key type lgssc unregistered [ 2889.777233] LNet: 147270:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2889.791769] LNetError: 147270:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2889.815608] LNet: Removed LNI 192.168.202.25@tcp [ 2890.760140] Key type .llcrypt unregistered [ 2890.761311] Key type ._llcrypt unregistered