[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 419488483 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001014] APIC: Switch to symmetric I/O mode setup [ 0.003268] x2apic enabled [ 0.004010] Switched APIC routing to physical x2apic. [ 0.005015] kvm-guest: setup PV IPIs [ 0.007697] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008037] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009035] pid_max: default: 32768 minimum: 301 [ 0.010149] LSM: Security Framework initializing [ 0.011054] Yama: becoming mindful. [ 0.012040] SELinux: Initializing. [ 0.013086] *** VALIDATE selinux *** [ 0.021658] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026080] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028087] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029130] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030126] *** VALIDATE tmpfs *** [ 0.032006] *** VALIDATE proc *** [ 0.033243] *** VALIDATE cgroup *** [ 0.034011] *** VALIDATE cgroup2 *** [ 0.035291] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036178] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037011] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.039026] Spectre V2 : User space: Vulnerable [ 0.040012] Speculative Store Bypass: Vulnerable [ 0.043028] debug: unmapping init [mem 0xffffffffb1459000-0xffffffffb1460fff] [ 0.045778] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.046743] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.047023] ... version: 2 [ 0.048015] ... bit width: 48 [ 0.049013] ... generic registers: 4 [ 0.050013] ... value mask: 0000ffffffffffff [ 0.051019] ... max period: 00007fffffffffff [ 0.052014] ... fixed-purpose events: 3 [ 0.053012] ... event mask: 000000070000000f [ 0.054308] rcu: Hierarchical SRCU implementation. [ 0.056493] smp: Bringing up secondary CPUs ... [ 0.057662] x86: Booting SMP configuration: [ 0.058030] .... node #0, CPUs: #1 #2 #3 [ 0.061317] smp: Brought up 1 node, 4 CPUs [ 0.063013] smpboot: Max logical packages: 1 [ 0.064019] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.196636] node 0 deferred pages initialised in 130ms [ 0.200210] devtmpfs: initialized [ 0.201224] x86/mm: Memory block size: 128MB [ 0.203525] gcov: version magic: 0x41383552 [ 0.204843] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.209156] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.211618] pinctrl core: initialized pinctrl subsystem [ 0.214231] [ 0.214930] ************************************************************* [ 0.217033] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.220027] ** ** [ 0.222016] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.224019] ** ** [ 0.227084] ** This means that this kernel is built to expose internal ** [ 0.229018] ** IOMMU data structures, which may compromise security on ** [ 0.232020] ** your system. ** [ 0.235020] ** ** [ 0.237016] ** If you see this message and you are not debugging the ** [ 0.240021] ** kernel, report this immediately to your vendor! ** [ 0.242015] ** ** [ 0.244016] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.245016] ************************************************************* [ 0.248945] NET: Registered protocol family 16 [ 0.250547] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.253081] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.256094] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.260076] cpuidle: using governor menu [ 0.262044] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.265864] PCI: Using configuration type 1 for base access [ 0.268164] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.278176] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.279101] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.281057] cryptd: max_cpu_qlen set to 1000 [ 0.284339] ACPI: Added _OSI(Module Device) [ 0.285014] ACPI: Added _OSI(Processor Device) [ 0.287021] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.288017] ACPI: Added _OSI(Processor Aggregator Device) [ 0.292204] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.298043] ACPI: Interpreter enabled [ 0.299000] ACPI: PM: (supports S0 S3 S4 S5) [ 0.299000] ACPI: Using IOAPIC for interrupt routing [ 0.301147] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.304456] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.314418] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.316048] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.318030] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.321118] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.327689] acpiphp: Slot [2] registered [ 0.329299] acpiphp: Slot [5] registered [ 0.330194] acpiphp: Slot [6] registered [ 0.332234] acpiphp: Slot [3] registered [ 0.333132] acpiphp: Slot [4] registered [ 0.335144] acpiphp: Slot [7] registered [ 0.336248] acpiphp: Slot [8] registered [ 0.338116] acpiphp: Slot [9] registered [ 0.339118] acpiphp: Slot [10] registered [ 0.340149] acpiphp: Slot [11] registered [ 0.342190] acpiphp: Slot [12] registered [ 0.343130] acpiphp: Slot [13] registered [ 0.344118] acpiphp: Slot [14] registered [ 0.346115] acpiphp: Slot [15] registered [ 0.347114] acpiphp: Slot [16] registered [ 0.348113] acpiphp: Slot [17] registered [ 0.349121] acpiphp: Slot [18] registered [ 0.350168] acpiphp: Slot [19] registered [ 0.351107] acpiphp: Slot [20] registered [ 0.353138] acpiphp: Slot [21] registered [ 0.354157] acpiphp: Slot [22] registered [ 0.355113] acpiphp: Slot [23] registered [ 0.356111] acpiphp: Slot [24] registered [ 0.358165] acpiphp: Slot [25] registered [ 0.359127] acpiphp: Slot [26] registered [ 0.362179] acpiphp: Slot [27] registered [ 0.363126] acpiphp: Slot [28] registered [ 0.365136] acpiphp: Slot [29] registered [ 0.366154] acpiphp: Slot [30] registered [ 0.368145] acpiphp: Slot [31] registered [ 0.370090] PCI host bridge to bus 0000:00 [ 0.372026] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.374031] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.376030] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.378032] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.380027] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.382035] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.385298] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.388000] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.390331] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.397459] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.400978] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.404026] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.405023] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.407022] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.409733] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.412798] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.415070] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.418094] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.421987] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.432642] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.436020] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.442337] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.447022] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.452018] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.461021] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.470425] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.476023] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.484025] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.497040] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.508086] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.511629] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.514407] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.516516] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.519293] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.525179] iommu: Default domain type: Passthrough [ 0.527499] SCSI subsystem initialized [ 0.529193] ACPI: bus type USB registered [ 0.531245] usbcore: registered new interface driver usbfs [ 0.533134] usbcore: registered new interface driver hub [ 0.534120] usbcore: registered new device driver usb [ 0.536208] pps_core: LinuxPPS API ver. 1 registered [ 0.538016] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.540114] PTP clock support registered [ 0.542127] EDAC MC: Ver: 3.0.0 [ 0.543388] PCI: Using ACPI for IRQ routing [ 0.545843] NetLabel: Initializing [ 0.547020] NetLabel: domain hash size = 128 [ 0.550019] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.552107] NetLabel: unlabeled traffic allowed by default [ 0.554170] vgaarb: loaded [ 0.556064] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.557013] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.566106] clocksource: Switched to clocksource kvm-clock [ 0.670949] VFS: Disk quotas dquot_6.6.0 [ 0.672647] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.674864] *** VALIDATE ramfs *** [ 0.676325] *** VALIDATE hugetlbfs *** [ 0.678096] pnp: PnP ACPI init [ 0.680695] pnp: PnP ACPI: found 6 devices [ 0.698069] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.700938] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.704492] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.706650] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.709501] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.712824] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.717074] NET: Registered protocol family 2 [ 0.719927] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.726220] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.730183] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.736130] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.740123] TCP: Hash tables configured (established 65536 bind 65536) [ 0.743203] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.746118] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.749204] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.752524] NET: Registered protocol family 1 [ 0.755411] RPC: Registered named UNIX socket transport module. [ 0.757817] RPC: Registered udp transport module. [ 0.759715] RPC: Registered tcp transport module. [ 0.761499] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.763816] NET: Registered protocol family 44 [ 0.765622] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.767940] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.769713] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.771464] PCI: CLS 0 bytes, default 64 [ 0.773164] Unpacking initramfs... [ 2.184991] debug: unmapping init [mem 0xffff8a07fcc64000-0xffff8a07fffcffff] [ 2.190519] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.192219] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.196341] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.697517] Initialise system trusted keyrings [ 2.699120] Key type blacklist registered [ 2.700878] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.709417] zbud: loaded [ 2.712483] *** VALIDATE nfs *** [ 2.713798] *** VALIDATE nfs4 *** [ 2.715125] pstore: using deflate compression [ 2.717609] Platform Keyring initialized [ 2.827737] NET: Registered protocol family 38 [ 2.830619] Key type asymmetric registered [ 2.832677] Asymmetric key parser 'x509' registered [ 2.835054] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.837521] io scheduler mq-deadline registered [ 2.839589] io scheduler kyber registered [ 2.841393] io scheduler bfq registered [ 2.843554] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.846906] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.850364] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.853454] ACPI: Power Button [PWRF] [ 2.862282] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.870102] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.879394] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.903700] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.932366] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.937271] Non-volatile memory driver v1.3 [ 2.939157] Linux agpgart interface v0.103 [ 2.973188] virtio_blk virtio1: [vda] 145896 512-byte logical blocks (74.7 MB/71.2 MiB) [ 2.975651] vda: detected capacity change from 0 to 74698752 [ 2.989773] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 2.992250] vdb: detected capacity change from 0 to 1073741824 [ 2.997561] libphy: Fixed MDIO Bus: probed [ 3.031960] usbcore: registered new interface driver usbserial_generic [ 3.034774] usbserial: USB Serial support registered for generic [ 3.037413] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.041725] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.043653] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.046038] mousedev: PS/2 mouse device common for all mice [ 3.048371] rtc_cmos 00:05: RTC can wake from S4 [ 3.051194] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.052809] rtc_cmos 00:05: registered as rtc0 [ 3.055620] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.056188] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.060606] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.062851] intel_pstate: CPU model not supported [ 3.067451] hid: raw HID events driver (C) Jiri Kosina [ 3.069477] usbcore: registered new interface driver usbhid [ 3.071507] usbhid: USB HID core driver [ 3.073186] drop_monitor: Initializing network drop monitor service [ 3.075612] Initializing XFRM netlink socket [ 3.077530] NET: Registered protocol family 10 [ 3.080643] Segment Routing with IPv6 [ 3.081904] NET: Registered protocol family 17 [ 3.083493] mpls_gso: MPLS GSO support [ 3.088660] RAS: Correctable Errors collector initialized. [ 3.090235] AVX version of gcm_enc/dec engaged. [ 3.091800] AES CTR mode by8 optimization enabled [ 3.166038] sched_clock: Marking stable (3166007265, 0)->(3964021700, -798014435) [ 3.170864] registered taskstats version 1 [ 3.173083] Loading compiled-in X.509 certificates [ 3.174860] zswap: loaded using pool lzo/zbud [ 3.201397] Key type big_key registered [ 3.215261] Key type encrypted registered [ 3.217237] ima: No TPM chip found, activating TPM-bypass! [ 3.219602] ima: Allocated hash algorithm: sha1 [ 3.221569] ima: No architecture policies found [ 3.223615] evm: Initialising EVM extended attributes: [ 3.225756] evm: security.selinux [ 3.227163] evm: security.ima [ 3.228444] evm: security.capability [ 3.229969] evm: HMAC attrs: 0x1 [ 3.232806] rtc_cmos 00:05: setting system clock to 2026-08-08 06:05:08 UTC (1786169108) [ 3.240934] debug: unmapping init [mem 0xffffffffb2403000-0xffffffffb25fffff] [ 3.245157] debug: unmapping init [mem 0xffffffffb1182000-0xffffffffb1458fff] [ 3.252269] Write protecting the kernel read-only data: 28672k [ 3.255458] debug: unmapping init [mem 0xffffffffaf803000-0xffffffffaf9fffff] [ 3.257970] debug: unmapping init [mem 0xffffffffb0114000-0xffffffffb01fffff] [ 3.289258] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.301310] systemd[1]: Detected virtualization kvm. [ 3.303691] systemd[1]: Detected architecture x86-64. [ 3.306293] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.331823] systemd[1]: No hostname configured. [ 3.334576] systemd[1]: Set hostname to . [ 3.337403] random: systemd: uninitialized urandom read (16 bytes read) [ 3.340990] systemd[1]: Initializing machine ID from random generator. [ 3.508646] random: systemd: uninitialized urandom read (16 bytes read) [ 3.510986] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 3.514710] random: systemd: uninitialized urandom read (16 bytes read) [ 3.517624] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 3.522385] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on Journal Socket (/dev/log). Starting Create Volatile Files and Directories... [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Timers. Starting Apply Kernel Variables... [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. Starting Journal Service... [ OK ] Reached target Swap. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. Starting Setup Virtual Console... [ OK ] Reached target Initrd Root Device. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.245799] device-mapper: uevent: version 1.0.3 [ 4.247862] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ 4.583844] random: fast init done [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 4.991132] virtio_net virtio0 ens2: renamed from eth0 [ 5.053379] scsi host0: ata_piix [ 5.074522] scsi host1: ata_piix [ 5.076477] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 5.079066] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 8.836618] dracut-initqueue[587]: RTNETLINK answers: File exists [ 9.734936] random: crng init done [ 9.736386] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.196446] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.291986] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.547363] SELinux: Disabled at runtime. [ 11.606611] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.614971] systemd[1]: Detected virtualization kvm. [ 11.617294] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.186431] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.190905] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.195939] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.200320] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.204264] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.213788] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.219221] systemd[1]: proc-sys-fs-binfmt_misc.automount: Refusing to start, unit to trigger not loaded. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Starting Create list of required st…ce nodes for the current kernel... Mounting POSIX Message Queue File System... [ OK ] Created slice User and Session Slice. [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Created slice system-getty.slice. Activating swap /dev/disk/by-label/SWAP... Starting Remount Root and Kernel File Systems... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Stopped target Switch Root. [ OK ] Created slice system-serial\x2dgetty.slice. [ 12.335801] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Starting Apply Kernel Variables... [ OK ] Stopped target Initrd File Systems. [ OK ] Reached target Local Encrypted Volumes. Mounting Huge Pages File System... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on udev Control Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Slices. [ OK ] Reached target RPC Port Mapper. [ OK ] Reached target rpc_pipefs.target. Starting udev Coldplug all Devices... [ OK ] Stopped target Initrd Root File System. [ OK ] Created slice system-sshd\x2dkeygen.slice. Mounting Kernel Debug File System... [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 12.882198] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.276976] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.343992] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.491869] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.513504] EDAC sbridge: Ver: 1.1.2 [ 14.751683] Key type dns_resolver registered [ 15.077986] NFS: Registering the id_resolver key type [ 15.080182] Key type id_resolver registered [ 15.081881] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Load/Save Random Seed. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Login Service... [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg122-client login: [ 59.297552] libcfs: loading out-of-tree module taints kernel. [ 59.590251] Key type ._llcrypt registered [ 59.593324] Key type .llcrypt registered [ 60.225938] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 60.240904] alg: No test for adler32 (adler32-zlib) [ 61.614422] Lustre: Lustre: Build Version: 2.17.56_50_gc905d6b [ 62.282834] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 64.009432] Key type lgssc registered [ 64.199627] hrtimer: interrupt took 3697615 ns [ 65.896494] Lustre: Echo OBD driver; http://www.lustre.org/ [ 183.013077] Lustre: Mounted lustre-client - version 2.17.56_50_gc905d6b [ 188.071592] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 202.078538] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing check_logdir /tmp/testlogs/ [ 207.737431] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing yml_node [ 208.868439] Lustre: lustre-OST0000-osc-ffff8a084b805000: disconnect after 24s idle [ 212.546164] Lustre: DEBUG MARKER: Client: 2.17.56.50 [ 215.470812] Lustre: DEBUG MARKER: MDS: 2.17.56.50 [ 217.775751] Lustre: DEBUG MARKER: OSS: 2.17.56.50 [ 219.407303] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Sat Aug 8 02:08:43 EDT 2026 [ 233.676808] Lustre: Unmounted lustre-client [ 254.033571] Key type lgssc unregistered [ 254.236241] LNet: 8419:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 254.241935] LNetError: 8419:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 254.260211] LNet: Removed LNI 192.168.201.22@tcp [ 254.922408] Key type .llcrypt unregistered [ 254.926562] Key type ._llcrypt unregistered [ 278.354850] Key type ._llcrypt registered [ 278.357365] Key type .llcrypt registered [ 278.509454] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 278.521819] alg: No test for adler32 (adler32-zlib) [ 279.528537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 279.624339] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 279.685647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 284.771632] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 285.001810] LNet: 9307:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 285.006783] LNetError: 9307:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 285.018435] LNet: Removed LNI 192.168.201.22@tcp [ 285.512302] Key type .llcrypt unregistered [ 285.514988] Key type ._llcrypt unregistered [ 292.415547] Key type ._llcrypt registered [ 292.417471] Key type .llcrypt registered [ 294.011611] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 02:09:57 (1786169397) [ 294.504299] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 294.524489] alg: No test for adler32 (adler32-zlib) [ 295.484336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 295.571770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 295.583189] LNet: 9721:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 295.603607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 295.669526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 295.680437] LNet: 9728:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 303.371589] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 02:10:06 (1786169406) [ 303.656481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 303.708288] LNet: 10302:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 303.785925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 303.948590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 312.030623] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 02:10:15 (1786169415) [ 312.359085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 312.377208] LNet: 10891:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 312.393489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 312.461819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 318.260754] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 02:10:22 (1786169422) [ 318.547761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 318.573550] LNet: 11475:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 318.601296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 318.704878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 324.684679] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 02:10:28 (1786169428) [ 324.775114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 324.810564] LNet: 12052:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 324.885043] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 324.962501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 325.428828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 331.517753] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 02:10:35 (1786169435) [ 331.617520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 331.678417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 331.750162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 332.019195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 332.428334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 338.731362] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 02:10:42 (1786169442) [ 339.166422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 339.196611] LNet: 13246:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 339.205388] LNet: 13246:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 339.228835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 339.339306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 346.856280] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 02:10:50 (1786169450) [ 346.925081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 346.992456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 347.136303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 347.201715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 347.765830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 347.803550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 348.616851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 348.703869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 349.282175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 349.344682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 349.867714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 349.911708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 350.292366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 350.322986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 350.886881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 350.914518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 351.419523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 351.438438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 351.847593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 351.875799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 352.255115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 352.292598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 352.752508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 352.798625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 353.250640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 353.322405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 353.612200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 353.643494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 359.579424] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 02:11:03 (1786169463) [ 359.724341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 359.753197] LNet: 14514:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 359.762790] LNet: 14514:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 359.789994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 359.854954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 359.891477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 366.288731] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 02:11:10 (1786169470) [ 366.429255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 366.479869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 366.612445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 366.650496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 372.591261] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 02:11:16 (1786169476) [ 372.843913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 372.896422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 372.956699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 372.983984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 378.266787] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 02:11:22 (1786169482) [ 378.433121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 378.466616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 378.523700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 378.560068] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 384.140795] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 02:11:28 (1786169488) [ 384.497457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 384.543822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 384.678354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 384.729200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 390.958150] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 02:11:34 (1786169494) [ 391.044611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 391.097804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 391.153473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 391.185795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 397.146390] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 02:11:41 (1786169501) [ 397.549060] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 397.580393] LNet: 18034:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 397.588789] LNet: 18034:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 5 previous similar messages [ 397.629402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 397.771924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 398.183256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 403.890556] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 02:11:48 (1786169508) [ 404.274647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 404.328499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 404.592696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 404.842613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 411.146249] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 02:11:55 (1786169515) [ 411.233317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 411.288468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 411.434653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 411.672483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 418.889707] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 02:12:02 (1786169522) [ 419.081447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 419.160792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 419.965411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 420.306625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 427.570916] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 02:12:11 (1786169531) [ 427.697124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 427.767350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 427.882971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 427.935800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 434.230681] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 02:12:18 (1786169538) [ 434.426168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 434.480801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 434.588753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 434.635649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 439.872838] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 02:12:24 (1786169544) [ 440.015861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 440.068698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 440.120852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 445.319102] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 02:12:29 (1786169549) [ 445.533852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 445.565877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 445.631167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 451.151646] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 02:12:34 (1786169554) [ 451.251652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 451.304708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 451.418887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 451.927611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 459.044362] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 02:12:42 (1786169562) [ 459.191086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 459.277171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 459.412506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 459.477095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 467.046568] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 02:12:50 (1786169570) [ 467.213072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 467.242176] LNet: 24057:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 467.251949] LNet: 24057:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 9 previous similar messages [ 467.299507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 467.397422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 467.430354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 474.585829] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 02:12:58 (1786169578) [ 474.719853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 474.749598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 474.814213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 474.840937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 481.008259] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 02:13:04 (1786169584) [ 481.086575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 481.128114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 481.187963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 481.215229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 488.582155] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 02:13:11 (1786169591) [ 488.660096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 488.752941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 488.837062] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 488.911082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 488.932983] LNetError: 25831:0:(peer.c:2362:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 488.966373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 488.998871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 496.164470] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 02:13:19 (1786169599) [ 496.253534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 496.298063] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 496.413346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.4mWl2M0ywU [ 496.489725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.4mWl2M0ywU [ 496.550214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.4mWl2M0ywU [ 502.396775] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 02:13:26 (1786169606) [ 502.464607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 502.528332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 502.604301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 502.645436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 502.693889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 508.984956] Lustre: DEBUG MARKER: == sanity-lnet test 40: LNetAddPeer merges a pre-existing non-primary peer NID ========================================================== 02:13:32 (1786169612) [ 509.106365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 509.204518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 509.351326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 509.392582] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 510.671407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.22@tcp [ 512.187747] Lustre: Lustre: Build Version: 2.17.56_50_gc905d6b [ 514.104240] Key type lgssc registered [ 515.359314] Lustre: Echo OBD driver; http://www.lustre.org/ [ 519.038018] LNetError: 27589:0:(lib-move.c:2158:lnet_handle_find_routed_path()) peer 192.168.201.22@tcp99 has no available nets [ 521.258328] Key type lgssc unregistered [ 521.572736] LNetError: 28704:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 521.588711] LNet: Removed LNI 192.168.201.22@tcp [ 522.107596] Key type .llcrypt unregistered [ 522.111613] Key type ._llcrypt unregistered [ 527.809260] Key type ._llcrypt registered [ 527.811940] Key type .llcrypt registered [ 528.796358] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 02:13:52 (1786169632) [ 529.165624] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 529.184570] alg: No test for adler32 (adler32-zlib) [ 530.074257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 530.135168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 532.581436] LNet: Message forwarding will be enabled in 2s [ 535.649133] LNet: Message forwarding enabled [ 535.973618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 536.003320] LNet: Message forwarding will stop in 2s to 6s [ 540.444764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 541.037524] LNet: Message forwarding disabled [ 542.640345] LNet: Message forwarding will be enabled in 2s [ 546.150616] LNet: Message forwarding enabled [ 548.813595] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 02:14:12 (1786169652) [ 548.927620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 548.951640] LNet: 30107:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 550.120541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 550.234401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 555.722744] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 02:14:19 (1786169659) [ 555.947618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 555.973514] LNet: 30694:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 555.989928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 556.041795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 556.069893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 556.100530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 556.129282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 556.161961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 556.188815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 556.211377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 556.247874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 556.285406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 556.317634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 556.346667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 556.391770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 556.425986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 556.455332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 556.493746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 556.539852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 556.573995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 556.616493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 556.651599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 556.667614] LNetError: 30737:0:(router.c:625:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 562.317946] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 02:14:26 (1786169666) [ 562.469266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 562.506305] LNet: 31304:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 562.546372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 562.639458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 568.880781] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 02:14:32 (1786169672) [ 571.162318] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg122-server [ 573.066786] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 02:14:36 (1786169676) [ 573.247970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 573.274093] LNet: 32238:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 573.305833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 573.356739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 573.386055] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 574.124879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.201.[23-27]@tcp [ 574.140447] LNet: 32371:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 574.180441] LNetError: 32242:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.201.24@tcp is being used as a gateway but routing feature is not turned on [ 574.201307] LNetError: 32242:0:(lib-lnet.h:1274:lnet_set_route_aliveness()) route to tcp1 through 192.168.201.24@tcp has gone from up to down [ 575.204892] LNetError: 32242:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.201.25@tcp is being used as a gateway but routing feature is not turned on [ 575.218305] LNetError: 32242:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 576.239193] LNetError: 32242:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.201.27@tcp is being used as a gateway but routing feature is not turned on [ 576.246674] LNetError: 32242:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 4 previous similar messages [ 578.276523] LNetError: 32242:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.201.27@tcp is being used as a gateway but routing feature is not turned on [ 578.284586] LNetError: 32242:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 579.964181] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 02:14:44 (1786169684) [ 581.742404] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg122-server [ 582.374700] LNetError: 32242:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.201.26@tcp is being used as a gateway but routing feature is not turned on [ 582.386730] LNetError: 32242:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 19 previous similar messages [ 583.532466] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 02:14:47 (1786169687) [ 583.728533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 583.748185] LNet: 33307:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 583.757374] LNetError: 33307:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 583.781951] LNet: Removed LNI 192.168.201.22@tcp [ 583.820394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 583.891117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 583.936201] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 584.585820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.201.[23-28/2]@tcp [ 584.626441] LNet: 33424:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 584.633699] LNet: 33424:0:(router.c:729:lnet_add_route()) Skipped 4 previous similar messages [ 584.661534] LNetError: 33311:0:(lib-lnet.h:1274:lnet_set_route_aliveness()) route to tcp103 through 192.168.201.27@tcp has gone from up to down [ 584.666392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.201.[23-28/2]@tcp [ 584.668709] LNetError: 33311:0:(lib-lnet.h:1274:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 590.227583] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 02:14:54 (1786169694) [ 590.490267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 590.514439] LNetError: 34016:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 590.535397] LNet: Removed LNI 192.168.201.22@tcp [ 590.552802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 590.675255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 590.761928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 590.802107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 590.870465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 590.912275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 590.950385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 590.983550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 591.035919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 591.067544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 591.156769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 591.241119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 591.264352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 591.307274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 591.407525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 591.454102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 591.531841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 591.681272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 591.735662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 591.788064] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 591.853590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 591.874267] LNet: 34110:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 591.878640] LNet: 34110:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 7 previous similar messages [ 591.901596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 591.948705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 592.052891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 592.104150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 592.158710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 592.249892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 592.347452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 592.452567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 592.553946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 592.581868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 592.625959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 592.676199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 592.708321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 592.788894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 598.572810] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 02:15:02 (1786169702) [ 600.121777] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg122-server [ 601.647132] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 02:15:05 (1786169705) [ 603.270637] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg122-server [ 604.876852] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 02:15:08 (1786169708) [ 605.892070] Key type .llcrypt unregistered [ 605.895367] Key type ._llcrypt unregistered [ 611.445842] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 611.593833] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 611.597311] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 611.602727] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 614.211990] Key type ._llcrypt registered [ 614.213833] Key type .llcrypt registered [ 614.339465] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 614.352223] alg: No test for adler32 (adler32-zlib) [ 615.241623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 615.282405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 615.307819] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 615.801851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 615.819887] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 617.538137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 617.546943] LNet: Removed LNI 10.1.2.3@tcp [ 617.595576] LNet: 36144:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 617.597758] LNetError: 36144:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 617.606592] LNet: Removed LNI 192.168.201.22@tcp [ 618.024675] Key type .llcrypt unregistered [ 618.030814] Key type ._llcrypt unregistered [ 629.151191] Key type ._llcrypt registered [ 629.154735] Key type .llcrypt registered [ 630.284504] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 02:15:34 (1786169734) [ 630.978530] Key type .llcrypt unregistered [ 630.987344] Key type ._llcrypt unregistered [ 637.055084] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 637.162322] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 637.170372] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 637.172786] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 639.784724] Key type ._llcrypt registered [ 639.787525] Key type .llcrypt registered [ 639.942388] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 639.959691] alg: No test for adler32 (adler32-zlib) [ 640.890353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 640.943941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 640.960990] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 641.478303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 641.500325] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 643.475622] LNet: 37647:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 643.482654] LNetError: 37647:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 643.494355] LNet: Removed LNI 10.1.2.3@tcp [ 643.511497] LNet: Removed LNI 192.168.201.22@tcp [ 644.160775] Key type .llcrypt unregistered [ 644.163847] Key type ._llcrypt unregistered [ 655.600455] Key type ._llcrypt registered [ 655.601826] Key type .llcrypt registered [ 656.794587] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 02:16:00 (1786169760) [ 657.454391] Key type .llcrypt unregistered [ 657.456995] Key type ._llcrypt unregistered [ 663.035568] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 663.216977] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 663.230908] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 663.243199] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 664.761634] Key type ._llcrypt registered [ 664.764263] Key type .llcrypt registered [ 664.937861] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 664.954514] alg: No test for adler32 (adler32-zlib) [ 665.847459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 665.892529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 665.913102] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 667.530443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 667.546933] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 669.751022] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 669.793927] LNet: Removed LNI 10.1.2.3@tcp [ 669.817461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 669.853276] LNet: Removed LNI 10.1.2.31@tcp [ 669.867464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 669.909848] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 670.866070] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 670.893627] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 671.766379] LNet: 39564:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 671.778206] LNetError: 39564:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg:0 not found [ 671.788217] LNet: Removed LNI 10.1.2.31@tcp [ 671.804422] LNet: Removed LNI 10.1.2.3@tcp [ 672.406527] Key type .llcrypt unregistered [ 672.408769] Key type ._llcrypt unregistered [ 684.934411] Key type ._llcrypt registered [ 684.937814] Key type .llcrypt registered [ 686.356966] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 02:16:30 (1786169790) [ 686.675937] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 686.687536] alg: No test for adler32 (adler32-zlib) [ 687.627146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 687.668257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 687.692250] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 688.285853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 688.315145] LNet: 40521:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 688.321347] LNetError: 40521:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 688.358257] LNet: Removed LNI 192.168.201.22@tcp [ 688.382283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 688.427194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.201.22@tcp [ 688.450922] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 689.031362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.201.22@tcp [ 689.051690] LNet: Removed LNI 192.168.201.22@tcp [ 689.124439] LNet: 40656:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 689.569659] Key type .llcrypt unregistered [ 689.573536] Key type ._llcrypt unregistered [ 699.539657] Key type ._llcrypt registered [ 699.544154] Key type .llcrypt registered [ 700.820549] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 02:16:44 (1786169804) [ 702.368270] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg122-server [ 704.174415] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 02:16:47 (1786169807) [ 704.800480] Key type .llcrypt unregistered [ 704.804375] Key type ._llcrypt unregistered [ 711.405614] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 711.613767] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 711.623650] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 711.638967] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 714.337162] Key type ._llcrypt registered [ 714.338769] Key type .llcrypt registered [ 714.501346] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 714.520579] alg: No test for adler32 (adler32-zlib) [ 715.540524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 715.631783] LNet: Added LNI 192.168.201.22@tcp1 [8/256/0/180] [ 716.516646] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 718.976496] LNet: 42501:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 718.990112] LNetError: 42501:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 719.010901] LNet: Removed LNI 192.168.201.22@tcp1 [ 719.030912] LNet: Removed LNI 10.1.2.3@tcp2 [ 719.767537] Key type .llcrypt unregistered [ 719.769510] Key type ._llcrypt unregistered [ 732.549643] Key type ._llcrypt registered [ 732.551983] Key type .llcrypt registered [ 734.137889] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 02:17:17 (1786169837) [ 734.703301] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 734.717637] alg: No test for adler32 (adler32-zlib) [ 735.667557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 735.725391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 735.769089] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 736.633229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 736.655814] LNet: 43477:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 736.662591] LNetError: 43477:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 736.679038] LNet: Removed LNI 192.168.201.22@tcp [ 736.725830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 736.827354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 736.872696] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 744.467670] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 02:17:28 (1786169848) [ 744.605682] LNet: 44060:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 744.614782] LNetError: 44060:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 744.637901] LNet: Removed LNI 192.168.201.22@tcp [ 745.396532] Key type .llcrypt unregistered [ 745.398557] Key type ._llcrypt unregistered [ 752.638173] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 752.860669] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 752.869184] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 752.872615] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 755.614349] Key type ._llcrypt registered [ 755.616767] Key type .llcrypt registered [ 755.837546] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 755.850846] alg: No test for adler32 (adler32-zlib) [ 756.829778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 756.907626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 756.939324] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 757.797207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 757.817045] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 759.941656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 759.955571] LNet: 44815:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 759.959847] LNetError: 44815:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 759.965370] LNet: Removed LNI 10.1.2.3@tcp [ 759.990608] LNet: Removed LNI 192.168.201.22@tcp [ 760.024512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 760.076756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 760.096758] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 760.100453] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 760.765474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 760.809558] LNet: 44843:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 760.819602] LNetError: 44843:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 760.829870] LNetError: 44843:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 760.838189] LNet: Removed LNI 10.1.2.3@tcp [ 760.859359] LNet: Removed LNI 192.168.201.22@tcp [ 760.901838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 761.001356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 761.026254] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 761.030192] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 761.283319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 761.298908] LNet: Added LNI 192.168.201.22@tcp2 [8/256/0/180] [ 762.441167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 762.465897] LNet: 45005:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 762.470847] LNetError: 45005:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 762.477640] LNetError: 45005:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 762.482934] LNet: Removed LNI 10.1.2.3@tcp [ 762.485155] LNet: Removed LNI 192.168.201.22@tcp [ 762.504819] LNet: Removed LNI 192.168.201.22@tcp2 [ 762.525868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 762.592529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 762.615200] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 762.625219] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 762.629226] LNet: Added LNI 192.168.201.22@tcp2 [8/256/0/180] [ 769.701618] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 02:17:53 (1786169873) [ 769.773294] LNet: 45595:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 769.786796] LNetError: 45595:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 769.792266] LNetError: 45595:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 769.806810] LNet: Removed LNI 10.1.2.3@tcp [ 769.809840] LNet: Removed LNI 192.168.201.22@tcp [ 769.823173] LNet: Removed LNI 192.168.201.22@tcp2 [ 770.598658] Key type .llcrypt unregistered [ 770.602731] Key type ._llcrypt unregistered [ 777.495815] Key type ._llcrypt registered [ 777.497657] Key type .llcrypt registered [ 777.772671] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 777.797564] alg: No test for adler32 (adler32-zlib) [ 778.796932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 778.870973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 778.902292] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 779.760695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 779.780374] LNet: 46003:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 779.796302] LNetError: 46003:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 779.811996] LNet: Removed LNI 192.168.201.22@tcp [ 779.840611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 779.965235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 779.997859] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 780.762438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 780.798870] LNet: 46034:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 780.805282] LNetError: 46034:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 780.817890] LNet: Removed LNI 192.168.201.22@tcp [ 780.843764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 780.895737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 780.940273] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 781.456107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 781.505863] LNet: Removed LNI 192.168.201.22@tcp [ 781.528347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 781.617591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 781.636220] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 781.908024] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 782.016558] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 782.029499] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 782.045477] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 784.552261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 784.608301] LNet: 46142:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 784.620853] LNet: 46142:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 784.642218] LNetError: 46142:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 784.646762] LNetError: 46142:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 784.663863] LNet: Removed LNI 192.168.201.22@tcp [ 784.679818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 784.739375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 784.773988] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 791.612736] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 02:18:15 (1786169895) [ 791.853501] LNet: 46735:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 791.864372] LNetError: 46735:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 791.911056] LNet: Removed LNI 192.168.201.22@tcp [ 792.766094] Key type .llcrypt unregistered [ 792.767728] Key type ._llcrypt unregistered [ 800.446343] Key type ._llcrypt registered [ 800.449853] Key type .llcrypt registered [ 800.663338] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 800.682530] alg: No test for adler32 (adler32-zlib) [ 801.570598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 801.603658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 801.621418] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 802.410500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 802.434501] LNet: 47143:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 802.439457] LNetError: 47143:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 802.453895] LNet: Removed LNI 192.168.201.22@tcp [ 802.475269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 802.521162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 802.543611] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 808.787291] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 02:18:32 (1786169912) [ 808.912631] LNet: 47727:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 808.917553] LNetError: 47727:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 808.937429] LNet: Removed LNI 192.168.201.22@tcp [ 809.685455] Key type .llcrypt unregistered [ 809.687366] Key type ._llcrypt unregistered [ 815.865938] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 816.078990] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 816.108278] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 816.115979] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 818.646355] Key type ._llcrypt registered [ 818.652365] Key type .llcrypt registered [ 818.864299] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 818.876341] alg: No test for adler32 (adler32-zlib) [ 819.855530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 819.915567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 819.950145] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 820.743637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 820.758033] LNet: 48192:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 820.762552] LNetError: 48192:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 820.771762] LNet: Removed LNI 192.168.201.22@tcp [ 820.784595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 820.827589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 820.846641] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 826.939308] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 02:18:50 (1786169930) [ 827.006655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 827.038914] LNet: 48783:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 827.045079] LNetError: 48783:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 827.070961] LNet: Removed LNI 192.168.201.22@tcp [ 827.098126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 827.244800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 833.400166] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 02:18:57 (1786169937) [ 833.510066] LNet: 49365:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 834.317775] Key type .llcrypt unregistered [ 834.320937] Key type ._llcrypt unregistered [ 840.436491] Key type ._llcrypt registered [ 840.438668] Key type .llcrypt registered [ 840.577898] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 840.592550] alg: No test for adler32 (adler32-zlib) [ 841.518389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 841.571466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 841.592520] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 842.399619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 842.425968] LNet: 49774:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 842.435459] LNetError: 49774:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 842.452980] LNet: Removed LNI 192.168.201.22@tcp [ 842.471526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 842.551603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 842.600209] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 849.024109] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 02:19:12 (1786169952) [ 849.509514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 849.550343] LNet: 50367:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 849.555698] LNetError: 50367:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 849.575419] LNet: Removed LNI 192.168.201.22@tcp [ 849.613769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 849.715683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 854.840550] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 02:19:19 (1786169959) [ 856.198188] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 858.128070] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 02:19:21 (1786169961) [ 859.458848] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 860.882643] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 02:19:24 (1786169964) [ 862.413424] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 864.158504] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 02:19:28 (1786169968) [ 865.732269] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 867.346372] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 02:19:31 (1786169971) [ 868.717580] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 870.010771] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing ========================================================== 02:19:34 (1786169974) [ 870.267461] LNet: 52419:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 870.851422] Key type .llcrypt unregistered [ 870.853254] Key type ._llcrypt unregistered [ 877.397228] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 877.570932] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 877.576204] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 877.582824] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 880.246878] Key type ._llcrypt registered [ 880.249580] Key type .llcrypt registered [ 880.525340] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 880.543771] alg: No test for adler32 (adler32-zlib) [ 881.589986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 881.673696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 881.708221] LNet: Added LNI 192.168.201.22@tcp [8/512/0/180] [ 881.712781] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 881.714491] LNet: 52743:0:(api-ni.c:2985:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 881.811646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 881.827558] LNet: 52762:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 881.834922] LNetError: 52762:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 881.844067] LNet: Removed LNI 10.1.2.3@tcp [ 881.845647] LNetError: 52762:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 881.863251] LNet: Removed LNI 192.168.201.22@tcp [ 881.889982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 881.933933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 881.955985] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 882.024983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 882.049928] LNet: 52786:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 882.055434] LNetError: 52786:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 882.085535] LNet: Removed LNI 192.168.201.22@tcp [ 882.119850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 882.237155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 882.263365] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 882.270913] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 882.272870] LNet: 52794:0:(api-ni.c:2985:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 882.400121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 882.417669] LNet: 52812:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 882.422674] LNetError: 52812:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 882.428939] LNet: Removed LNI 10.1.2.3@tcp [ 882.431303] LNetError: 52812:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 882.445011] LNet: Removed LNI 192.168.201.22@tcp [ 882.459641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 882.538678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 882.562506] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 882.571251] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 882.576875] LNet: 52820:0:(api-ni.c:2985:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 882.817586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 882.842580] LNet: 52846:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 882.849785] LNetError: 52846:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 882.852861] LNet: Removed LNI 10.1.2.3@tcp [ 882.866183] LNetError: 52846:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 882.879932] LNet: Removed LNI 192.168.201.22@tcp [ 882.905664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 882.968571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 882.993500] LNet: Added LNI 192.168.201.22@tcp [8/256/0/182] [ 882.998752] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 883.100429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 883.118370] LNet: 52874:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 883.125846] LNetError: 52874:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 883.133581] LNet: Removed LNI 10.1.2.3@tcp [ 883.135249] LNetError: 52874:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 883.145816] LNet: Removed LNI 192.168.201.22@tcp [ 883.159912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 883.248198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 883.269537] LNet: Added LNI 192.168.201.22@tcp [8/256/0/182] [ 883.275188] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 883.278179] LNet: 52883:0:(api-ni.c:2985:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 883.389986] LNet: Removed LNI 10.1.2.3@tcp [ 883.432239] LNet: Removed LNI 192.168.201.22@tcp [ 884.228112] Key type .llcrypt unregistered [ 884.237896] Key type ._llcrypt unregistered [ 895.979547] Key type ._llcrypt registered [ 895.982273] Key type .llcrypt registered [ 897.322474] Lustre: DEBUG MARKER: == sanity-lnet test 180: FastReg map error should not leak descriptors ========================================================== 02:20:00 (1786170000) [ 898.931216] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_180 Need o2ib NETTYPE [ 900.598308] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 02:20:04 (1786170004) [ 901.027135] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 901.038881] alg: No test for adler32 (adler32-zlib) [ 901.966664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 902.043975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 902.067885] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 902.604248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.201.22 [ 902.629914] LNet: 54153:0:(config.c:1738:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 902.639050] LNet: Removed LNI 192.168.201.22@tcp [ 908.600140] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 02:20:12 (1786170012) [ 908.761283] LNet: 54722:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 908.772414] LNetError: 54722:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 908.797798] LNet: Removed LNI 192.168.201.22@tcp [ 909.411269] Key type .llcrypt unregistered [ 909.412761] Key type ._llcrypt unregistered [ 916.295743] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 916.531953] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 919.265493] Key type ._llcrypt registered [ 919.267703] Key type .llcrypt registered [ 919.512737] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 919.532453] alg: No test for adler32 (adler32-zlib) [ 920.859413] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 927.811669] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 02:20:31 (1786170031) [ 927.983858] LNet: 55636:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 927.993685] LNetError: 55636:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 928.034333] LNet: Removed LNI 10.1.2.3@tcp [ 928.829939] Key type .llcrypt unregistered [ 928.832319] Key type ._llcrypt unregistered [ 935.139709] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 935.353553] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 935.359654] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 935.366711] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 938.169028] Key type ._llcrypt registered [ 938.176196] Key type .llcrypt registered [ 938.412407] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 938.436905] alg: No test for adler32 (adler32-zlib) [ 939.428197] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 945.505484] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 02:20:49 (1786170049) [ 946.931995] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 948.332880] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 02:20:52 (1786170052) [ 948.598275] LNet: 56832:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 948.610766] LNetError: 56832:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 948.630525] LNet: Removed LNI 10.1.2.3@tcp [ 949.366545] Key type .llcrypt unregistered [ 949.378934] Key type ._llcrypt unregistered [ 956.000385] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 956.302325] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 956.311213] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 956.322663] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 959.294548] Key type ._llcrypt registered [ 959.300655] Key type .llcrypt registered [ 959.631623] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 959.647447] alg: No test for adler32 (adler32-zlib) [ 960.641925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 960.740120] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 968.281515] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 02:21:12 (1786170072) [ 968.793908] LNet: 58065:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 968.799741] LNetError: 58065:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 968.820059] LNet: Removed LNI 10.1.2.3@tcp [ 969.476141] Key type .llcrypt unregistered [ 969.477951] Key type ._llcrypt unregistered [ 976.041680] Key type ._llcrypt registered [ 976.043818] Key type .llcrypt registered [ 976.337648] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 976.374546] alg: No test for adler32 (adler32-zlib) [ 977.339433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 977.376434] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 985.694409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.122@tcp [ 986.422266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.22@tcp -d 192.168.201.122@tcp -m GET -e local_interrupt -r 1 [ 986.465232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.22@tcp -d 192.168.201.22@tcp -m GET -e local_interrupt -r 1 [ 986.507501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.122@tcp [ 986.533938] LNet: There was an unexpected network error while writing to 192.168.201.122: rc = -22 [ 986.592121] LNet: 1 local NIs in recovery (showing 1): 192.168.201.22@tcp [ 986.761521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 986.785068] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 986.813908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 986.841829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 986.876511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 986.915425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 986.951510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 986.995684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 987.013851] LNet: 58507:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 987.035268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 987.070595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 987.104643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 987.134800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 987.161585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 987.199294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 987.237489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 987.869602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp -d 192.168.201.122@tcp -m GET -e local_dropped -r 1 [ 987.911450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.22@tcp -d 192.168.201.22@tcp -m GET -e local_dropped -r 1 [ 987.949979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.122@tcp [ 987.965193] LNet: There was an unexpected network error while writing to 192.168.201.122: rc = -22 [ 988.203467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 988.257056] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 988.280382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 988.303657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 988.326239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 988.348668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 988.371356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 988.392224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 988.404532] LNet: 58616:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 988.418265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 988.439955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 988.466666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 988.490833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 988.513825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 988.543841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 988.568651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 989.002315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.22@tcp -d 192.168.201.122@tcp -m GET -e local_aborted -r 1 [ 989.033863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp -d 192.168.201.22@tcp -m GET -e local_aborted -r 1 [ 989.062218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.122@tcp [ 989.082587] LNet: There was an unexpected network error while writing to 192.168.201.122: rc = -22 [ 989.329588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 989.373325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 989.408449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 989.453391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 989.505633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 989.543987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 989.572478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 989.605608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 989.633173] LNet: 58725:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 989.661890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 989.690953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 989.719156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 989.752424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 989.777566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 989.815796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 989.853992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 990.367498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.22@tcp -d 192.168.201.122@tcp -m GET -e local_no_route -r 1 [ 990.414387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.22@tcp -d 192.168.201.22@tcp -m GET -e local_no_route -r 1 [ 990.445378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.122@tcp [ 990.674368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 990.699280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 990.724463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 990.751481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 990.776980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 990.803724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 990.838309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 990.864184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 990.899846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 990.928428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 990.962326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 990.988430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 991.009450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 991.040355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 991.074851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 991.686681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.22@tcp -d 192.168.201.122@tcp -m GET -e local_timeout -r 1 [ 991.718475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.22@tcp -d 192.168.201.22@tcp -m GET -e local_timeout -r 1 [ 991.749047] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.122@tcp [ 991.784807] LNet: There was an unexpected network error while writing to 192.168.201.122: rc = -22 [ 991.787841] LNet: Skipped 2 previous similar messages [ 992.084582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 992.123367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 992.156712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 992.179882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 992.201076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 992.219462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 992.249785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 992.269920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 992.282711] LNet: 58944:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 992.286804] LNet: 58944:0:(api-ni.c:295:recovery_interval_set()) Skipped 1 previous similar message [ 992.297412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 992.330824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 992.368513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 992.400254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 992.459139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 992.507189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 992.536926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 993.077944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp -d 192.168.201.122@tcp -m GET -e local_error -r 1 [ 993.110761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.22@tcp -d 192.168.201.22@tcp -m GET -e local_error -r 1 [ 993.140730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.122@tcp [ 993.405181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 993.446842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 993.488604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 993.522533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 993.569962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 993.605880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 993.653130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 993.705688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 993.781868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 993.838354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 993.914789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 993.973509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 994.017357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 994.063139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 994.103200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 994.374258] LNet: 59098:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 994.381413] LNetError: 59098:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 994.392980] LNet: Removed LNI 192.168.201.22@tcp [ 994.863507] Key type .llcrypt unregistered [ 994.865588] Key type ._llcrypt unregistered [ 1011.842207] Key type ._llcrypt registered [ 1011.845115] Key type .llcrypt registered [ 1012.940681] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 02:21:56 (1786170116) [ 1013.777532] Key type .llcrypt unregistered [ 1013.782407] Key type ._llcrypt unregistered [ 1019.925620] Key type ._llcrypt registered [ 1019.927204] Key type .llcrypt registered [ 1020.085941] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1020.097617] alg: No test for adler32 (adler32-zlib) [ 1021.070911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1021.093115] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 1029.567975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.122@tcp [ 1040.761152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1040.811264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1040.843731] LNet: Added LNI 192.168.201.22@tcp1 [8/256/0/180] [ 1042.116748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp -d 192.168.201.122@tcp -m GET -e local_interrupt -r 1 [ 1042.153378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.22@tcp -d 192.168.201.122@tcp1 -m GET -e local_interrupt -r 1 [ 1042.191745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp -d 192.168.201.22@tcp -m GET -e local_interrupt -r 1 [ 1042.230641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp -d 192.168.201.22@tcp1 -m GET -e local_interrupt -r 1 [ 1042.270371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp1 -d 192.168.201.122@tcp -m GET -e local_interrupt -r 1 [ 1042.314538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp1 -d 192.168.201.122@tcp1 -m GET -e local_interrupt -r 1 [ 1042.349939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp1 -d 192.168.201.22@tcp -m GET -e local_interrupt -r 1 [ 1042.402245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp1 -d 192.168.201.22@tcp1 -m GET -e local_interrupt -r 1 [ 1042.440760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.122@tcp [ 1042.455600] LNet: There was an unexpected network error while writing to 192.168.201.122: rc = -22 [ 1042.459895] LNet: 1 local NIs in recovery (showing 1): 192.168.201.22@tcp [ 1042.689610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1042.724385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1042.763912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1042.789370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1042.817708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1042.839881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1042.870711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1042.899829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1042.928899] LNet: 60588:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1042.946985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1042.980727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1043.004443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1043.043593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1043.079266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1043.116273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1043.163065] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1043.572864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp -d 192.168.201.122@tcp -m GET -e local_dropped -r 1 [ 1043.613406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.22@tcp -d 192.168.201.122@tcp1 -m GET -e local_dropped -r 1 [ 1043.663358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.22@tcp -d 192.168.201.22@tcp -m GET -e local_dropped -r 1 [ 1043.713622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.22@tcp -d 192.168.201.22@tcp1 -m GET -e local_dropped -r 1 [ 1043.747851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.22@tcp1 -d 192.168.201.122@tcp -m GET -e local_dropped -r 1 [ 1043.780933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp1 -d 192.168.201.122@tcp1 -m GET -e local_dropped -r 1 [ 1043.812040] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp1 -d 192.168.201.22@tcp -m GET -e local_dropped -r 1 [ 1043.845742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp1 -d 192.168.201.22@tcp1 -m GET -e local_dropped -r 1 [ 1043.883505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.122@tcp [ 1043.912526] LNet: There was an unexpected network error while writing to 192.168.201.122: rc = -22 [ 1043.916755] LNet: Skipped 2 previous similar messages [ 1044.226261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1044.291965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1044.356373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1044.388737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1044.435089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1044.458957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1044.498000] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1044.531540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1044.555309] LNet: 60709:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1044.599621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1044.656741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1044.685919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1044.720266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1044.753424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1044.781886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1044.812502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1045.367263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.22@tcp -d 192.168.201.122@tcp -m GET -e local_aborted -r 1 [ 1045.424192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp -d 192.168.201.122@tcp1 -m GET -e local_aborted -r 1 [ 1045.478881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp -d 192.168.201.22@tcp -m GET -e local_aborted -r 1 [ 1045.522667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp -d 192.168.201.22@tcp1 -m GET -e local_aborted -r 1 [ 1045.566654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.22@tcp1 -d 192.168.201.122@tcp -m GET -e local_aborted -r 1 [ 1045.609902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp1 -d 192.168.201.122@tcp1 -m GET -e local_aborted -r 1 [ 1045.670763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp1 -d 192.168.201.22@tcp -m GET -e local_aborted -r 1 [ 1045.725959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.22@tcp1 -d 192.168.201.22@tcp1 -m GET -e local_aborted -r 1 [ 1045.781502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.122@tcp [ 1045.818776] LNet: There was an unexpected network error while writing to 192.168.201.122: rc = -22 [ 1045.830314] LNet: Skipped 2 previous similar messages [ 1046.102778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1046.149664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1046.193633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1046.219878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1046.240334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1046.279567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1046.358514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1046.408538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1046.426592] LNet: 60830:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1046.461251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1046.494892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1046.545789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1046.574335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1046.615397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1046.673342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1046.736294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1047.251722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp -d 192.168.201.122@tcp -m GET -e local_no_route -r 1 [ 1047.281799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.22@tcp -d 192.168.201.122@tcp1 -m GET -e local_no_route -r 1 [ 1047.313661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp -d 192.168.201.22@tcp -m GET -e local_no_route -r 1 [ 1047.342567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.22@tcp -d 192.168.201.22@tcp1 -m GET -e local_no_route -r 1 [ 1047.377499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.22@tcp1 -d 192.168.201.122@tcp -m GET -e local_no_route -r 1 [ 1047.417431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp1 -d 192.168.201.122@tcp1 -m GET -e local_no_route -r 1 [ 1047.466814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp1 -d 192.168.201.22@tcp -m GET -e local_no_route -r 1 [ 1047.516414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.22@tcp1 -d 192.168.201.22@tcp1 -m GET -e local_no_route -r 1 [ 1047.566706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.122@tcp [ 1047.859879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1047.889606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1047.914279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1047.937764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1047.961584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1047.997618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1048.048463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1048.088404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1048.123891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1048.168894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1048.210144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1048.249922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1048.288393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1048.362332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1048.392867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1048.924202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp -d 192.168.201.122@tcp -m GET -e local_timeout -r 1 [ 1048.954221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.22@tcp -d 192.168.201.122@tcp1 -m GET -e local_timeout -r 1 [ 1048.986345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp -d 192.168.201.22@tcp -m GET -e local_timeout -r 1 [ 1049.012843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.22@tcp -d 192.168.201.22@tcp1 -m GET -e local_timeout -r 1 [ 1049.038907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp1 -d 192.168.201.122@tcp -m GET -e local_timeout -r 1 [ 1049.065541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.22@tcp1 -d 192.168.201.122@tcp1 -m GET -e local_timeout -r 1 [ 1049.107742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.22@tcp1 -d 192.168.201.22@tcp -m GET -e local_timeout -r 1 [ 1049.150956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp1 -d 192.168.201.22@tcp1 -m GET -e local_timeout -r 1 [ 1049.184958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.122@tcp [ 1049.203807] LNet: There was an unexpected network error while writing to 192.168.201.122: rc = -22 [ 1049.217855] LNet: Skipped 5 previous similar messages [ 1049.452815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1049.482905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1049.507332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1049.541837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1049.588568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1049.629396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1049.705700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1049.737085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1049.758123] LNet: 61073:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1049.765853] LNet: 61073:0:(api-ni.c:295:recovery_interval_set()) Skipped 1 previous similar message [ 1049.786365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1049.850581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1049.882880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1049.925949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1049.979554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1050.005885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1050.069487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1050.585721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp -d 192.168.201.122@tcp -m GET -e local_error -r 1 [ 1050.626249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp -d 192.168.201.122@tcp1 -m GET -e local_error -r 1 [ 1050.684547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp -d 192.168.201.22@tcp -m GET -e local_error -r 1 [ 1050.725241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.22@tcp -d 192.168.201.22@tcp1 -m GET -e local_error -r 1 [ 1050.751368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.22@tcp1 -d 192.168.201.122@tcp -m GET -e local_error -r 1 [ 1050.778952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.22@tcp1 -d 192.168.201.122@tcp1 -m GET -e local_error -r 1 [ 1050.802359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.22@tcp1 -d 192.168.201.22@tcp -m GET -e local_error -r 1 [ 1050.832563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.22@tcp1 -d 192.168.201.22@tcp1 -m GET -e local_error -r 1 [ 1050.858781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.122@tcp [ 1051.129648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1051.160085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1051.198428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1051.245583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1051.304412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1051.339880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1051.384257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1051.457043] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1051.495309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1051.558270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1051.601852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1051.643950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1051.681572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1051.720683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1051.762678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1057.286360] LNet: 61275:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1057.293752] LNetError: 61275:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1057.300161] LNet: Removed LNI 192.168.201.22@tcp [ 1058.352973] LNet: Removed LNI 192.168.201.22@tcp1 [ 1058.802786] Key type .llcrypt unregistered [ 1058.809777] Key type ._llcrypt unregistered [ 1074.244507] Key type ._llcrypt registered [ 1074.245867] Key type .llcrypt registered [ 1075.240524] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 02:22:59 (1786170179) [ 1076.066855] Key type .llcrypt unregistered [ 1076.077939] Key type ._llcrypt unregistered [ 1082.484228] Key type ._llcrypt registered [ 1082.486368] Key type .llcrypt registered [ 1082.645450] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1082.656409] alg: No test for adler32 (adler32-zlib) [ 1083.592275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1083.626946] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 1090.502976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.122@tcp [ 1091.355603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.22@tcp -d 192.168.201.122@tcp -m GET -e remote_dropped -r 1 [ 1091.409768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.22@tcp -d 192.168.201.22@tcp -m GET -e remote_dropped -r 1 [ 1091.458864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.122@tcp [ 1091.489140] LNet: There was an unexpected network error while writing to 192.168.201.122: rc = -22 [ 1091.796871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1091.830274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1091.857150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1091.888289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1091.929574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1091.971712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1092.008723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1092.048158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1092.072887] LNet: 62528:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1092.109141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1092.142170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1092.168412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1092.202163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1092.241509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1092.270376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1092.314717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1092.895246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp -d 192.168.201.122@tcp -m GET -e remote_error -r 1 [ 1092.929544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.22@tcp -d 192.168.201.22@tcp -m GET -e remote_error -r 1 [ 1092.966780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.122@tcp [ 1092.984794] LNet: There was an unexpected network error while writing to 192.168.201.122: rc = -22 [ 1093.275425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1093.301837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1093.324659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1093.352907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1093.385491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1093.415730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1093.455235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1093.497678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1093.518520] LNet: 62638:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1093.551988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1093.599920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1093.643784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1093.675599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1093.709240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1093.749824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1093.796782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1094.393139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.22@tcp -d 192.168.201.122@tcp -m GET -e remote_timeout -r 1 [ 1094.440666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.22@tcp -d 192.168.201.22@tcp -m GET -e remote_timeout -r 1 [ 1094.474986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.122@tcp [ 1094.495128] LNet: There was an unexpected network error while writing to 192.168.201.122: rc = -22 [ 1094.778230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1094.798346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1094.813210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1094.830600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1094.848716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1094.871785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1094.894821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1094.916991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1094.935240] LNet: 62747:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1094.947710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1094.968992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1094.989732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1095.012521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1095.042184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1095.067810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1095.093260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1095.431730] LNet: 62792:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1095.445055] LNetError: 62792:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1095.465483] LNet: Removed LNI 192.168.201.22@tcp [ 1095.865468] Key type .llcrypt unregistered [ 1095.867918] Key type ._llcrypt unregistered [ 1114.778775] Key type ._llcrypt registered [ 1114.781192] Key type .llcrypt registered [ 1116.148250] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 02:23:39 (1786170219) [ 1117.120702] Key type .llcrypt unregistered [ 1117.135601] Key type ._llcrypt unregistered [ 1124.838964] Key type ._llcrypt registered [ 1124.846137] Key type .llcrypt registered [ 1125.184594] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1125.228244] alg: No test for adler32 (adler32-zlib) [ 1126.157615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1126.181050] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 1133.312324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.122@tcp [ 1142.418821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1142.472383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1142.492384] LNet: Added LNI 192.168.201.22@tcp1 [8/256/0/180] [ 1143.908543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp -d 192.168.201.122@tcp -m GET -e remote_dropped -r 1 [ 1143.946408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.22@tcp -d 192.168.201.122@tcp1 -m GET -e remote_dropped -r 1 [ 1143.994501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.22@tcp -d 192.168.201.22@tcp -m GET -e remote_dropped -r 1 [ 1144.062517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.22@tcp -d 192.168.201.22@tcp1 -m GET -e remote_dropped -r 1 [ 1144.110915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.22@tcp1 -d 192.168.201.122@tcp -m GET -e remote_dropped -r 1 [ 1144.162580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp1 -d 192.168.201.122@tcp1 -m GET -e remote_dropped -r 1 [ 1144.216139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp1 -d 192.168.201.22@tcp -m GET -e remote_dropped -r 1 [ 1144.279529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.22@tcp1 -d 192.168.201.22@tcp1 -m GET -e remote_dropped -r 1 [ 1144.313252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.122@tcp [ 1144.332838] LNet: There was an unexpected network error while writing to 192.168.201.122: rc = -22 [ 1144.337552] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.122@tcp [ 1144.519412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1144.541422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1144.561963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1144.584965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1144.607279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1144.629928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1144.662391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1144.695160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1144.711558] LNet: 64266:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1144.725228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1144.753412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1144.775505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1144.803914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1144.833062] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1144.861476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1144.893234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1144.925720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1145.421322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp -d 192.168.201.122@tcp -m GET -e remote_error -r 1 [ 1145.454928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp -d 192.168.201.122@tcp1 -m GET -e remote_error -r 1 [ 1145.486684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.22@tcp -d 192.168.201.22@tcp -m GET -e remote_error -r 1 [ 1145.527136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp -d 192.168.201.22@tcp1 -m GET -e remote_error -r 1 [ 1145.589046] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp1 -d 192.168.201.122@tcp -m GET -e remote_error -r 1 [ 1145.610553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp1 -d 192.168.201.122@tcp1 -m GET -e remote_error -r 1 [ 1145.636590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.22@tcp1 -d 192.168.201.22@tcp -m GET -e remote_error -r 1 [ 1145.695261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp1 -d 192.168.201.22@tcp1 -m GET -e remote_error -r 1 [ 1145.747484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.122@tcp [ 1145.781197] LNet: There was an unexpected network error while writing to 192.168.201.122: rc = -22 [ 1145.786174] LNet: Skipped 2 previous similar messages [ 1146.082495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1146.120770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1146.162867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1146.234860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1146.273803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1146.328705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1146.377203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1146.440668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1146.485596] LNet: 64389:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1146.536434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1146.575837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1146.616566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1146.686158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1146.749661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1146.837712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1146.927808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1146.994847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1147.794309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.22@tcp -d 192.168.201.122@tcp -m GET -e remote_timeout -r 1 [ 1147.851831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.22@tcp -d 192.168.201.122@tcp1 -m GET -e remote_timeout -r 1 [ 1147.932665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.22@tcp -d 192.168.201.22@tcp -m GET -e remote_timeout -r 1 [ 1148.020268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.22@tcp -d 192.168.201.22@tcp1 -m GET -e remote_timeout -r 1 [ 1148.101922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp1 -d 192.168.201.122@tcp -m GET -e remote_timeout -r 1 [ 1148.147687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp1 -d 192.168.201.122@tcp1 -m GET -e remote_timeout -r 1 [ 1148.215393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.22@tcp1 -d 192.168.201.22@tcp -m GET -e remote_timeout -r 1 [ 1148.285599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp1 -d 192.168.201.22@tcp1 -m GET -e remote_timeout -r 1 [ 1148.328587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.122@tcp [ 1148.373500] LNet: There was an unexpected network error while writing to 192.168.201.122: rc = -22 [ 1148.390690] LNet: Skipped 1 previous similar message [ 1148.745121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1148.782767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1148.831286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1148.879770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1148.934948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1148.991942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1149.027794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1149.079526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1149.122426] LNet: 64512:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1149.165484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1149.240724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1149.313524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1149.354128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1149.402814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1149.439443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1149.480848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1149.539641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1155.050046] LNet: 64595:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1155.062270] LNetError: 64595:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1155.078969] LNet: Removed LNI 192.168.201.22@tcp [ 1155.136957] LNet: Removed LNI 192.168.201.22@tcp1 [ 1155.751542] Key type .llcrypt unregistered [ 1155.755069] Key type ._llcrypt unregistered [ 1177.811592] Key type ._llcrypt registered [ 1177.816361] Key type .llcrypt registered [ 1179.295488] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 02:24:42 (1786170282) [ 1181.138602] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 1183.323506] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 02:24:46 (1786170286) [ 1184.203543] Key type .llcrypt unregistered [ 1184.205973] Key type ._llcrypt unregistered [ 1189.869443] Key type ._llcrypt registered [ 1189.874800] Key type .llcrypt registered [ 1190.107293] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1190.125758] alg: No test for adler32 (adler32-zlib) [ 1191.039624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1191.068376] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 1198.403631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.122@tcp [ 1198.986896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp -d 192.168.201.122@tcp -m GET -e network_timeout -r 1 [ 1199.043707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp -d 192.168.201.22@tcp -m GET -e network_timeout -r 1 [ 1199.082349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.122@tcp [ 1199.107987] LNet: There was an unexpected network error while writing to 192.168.201.122: rc = -22 [ 1199.265800] LNet: 1 local NIs in recovery (showing 1): 192.168.201.22@tcp [ 1199.319635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1199.344878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1199.367184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1199.389603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1199.415757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1199.439670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1199.483374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1199.520162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1199.541688] LNet: 66145:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1199.576308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1199.626139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1199.659597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1199.694886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1199.749349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1199.790633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1200.050284] LNet: 66188:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1200.060253] LNetError: 66188:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1200.077203] LNet: Removed LNI 192.168.201.22@tcp [ 1200.548644] Key type .llcrypt unregistered [ 1200.562957] Key type ._llcrypt unregistered [ 1222.245949] Key type ._llcrypt registered [ 1222.248727] Key type .llcrypt registered [ 1222.420359] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1222.435892] alg: No test for adler32 (adler32-zlib) [ 1223.469493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1223.504843] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 1232.119450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.122@tcp [ 1241.632134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1241.674439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1241.700726] LNet: Added LNI 192.168.201.22@tcp1 [8/256/0/180] [ 1242.990342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.22@tcp -d 192.168.201.122@tcp -m GET -e network_timeout -r 1 [ 1243.030880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.22@tcp -d 192.168.201.122@tcp1 -m GET -e network_timeout -r 1 [ 1243.072080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.22@tcp -d 192.168.201.22@tcp -m GET -e network_timeout -r 1 [ 1243.120809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.22@tcp -d 192.168.201.22@tcp1 -m GET -e network_timeout -r 1 [ 1243.150108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.22@tcp1 -d 192.168.201.122@tcp -m GET -e network_timeout -r 1 [ 1243.181440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.22@tcp1 -d 192.168.201.122@tcp1 -m GET -e network_timeout -r 1 [ 1243.217596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp1 -d 192.168.201.22@tcp -m GET -e network_timeout -r 1 [ 1243.268377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.22@tcp1 -d 192.168.201.22@tcp1 -m GET -e network_timeout -r 1 [ 1243.302839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.122@tcp [ 1243.317615] LNet: There was an unexpected network error while writing to 192.168.201.122: rc = -22 [ 1243.545060] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1243.579952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1243.621854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1243.676577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1243.715116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1243.744916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1243.775783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1243.825065] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1243.865593] LNet: 67078:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1243.921835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1243.965586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1244.000097] LNet: 1 local NIs in recovery (showing 1): 192.168.201.22@tcp [ 1244.000556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1244.005376] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.122@tcp [ 1244.047655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1244.083030] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1244.115690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1248.450974] LNet: There was an unexpected network error while writing to 192.168.201.122: rc = -22 [ 1248.753193] LNet: 67156:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1248.759481] LNetError: 67156:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1249.824276] LNet: Removed LNI 192.168.201.22@tcp [ 1249.826868] LNetError: 67156:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1249.846872] LNet: Removed LNI 192.168.201.22@tcp1 [ 1250.360460] Key type .llcrypt unregistered [ 1250.367851] Key type ._llcrypt unregistered [ 1266.589782] Key type ._llcrypt registered [ 1266.591698] Key type .llcrypt registered [ 1267.823028] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 02:26:11 (1786170371) [ 1268.139699] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1268.155531] alg: No test for adler32 (adler32-zlib) [ 1269.039275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1269.087040] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1269.106961] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 1269.678212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1269.695512] LNet: Added LNI 192.168.201.22@tcp1 [8/256/0/180] [ 1270.463346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.22@tcp [ 1270.520735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1270.563321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1270.603843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1270.631192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1270.657714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1270.684414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.201.22@tcp [ 1270.702464] LNetError: 68256:0:(api-ni.c:4472:lnet_ni_set_healthv()) manually adding local NI 192.168.201.22@tcp to recovery [ 1271.136530] LNet: 1 local NIs in recovery (showing 1): 192.168.201.22@tcp [ 1271.145818] LNet: There was an unexpected network error while writing to 192.168.201.22: rc = -22 [ 1273.184539] LNet: There was an unexpected network error while writing to 192.168.201.22: rc = -22 [ 1274.289851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1277.282446] LNet: There was an unexpected network error while writing to 192.168.201.22: rc = -22 [ 1277.983574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1278.010966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1278.068527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1278.087977] LNet: 68363:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1278.095231] LNetError: 68363:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1279.136217] LNet: Removed LNI 192.168.201.22@tcp [ 1279.138586] LNetError: 68363:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1279.167933] LNet: Removed LNI 192.168.201.22@tcp1 [ 1279.238297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1279.308518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1279.356765] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 1280.477431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1280.495151] LNet: Added LNI 192.168.201.22@tcp1 [8/256/0/180] [ 1281.370242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.22@tcp [ 1281.434515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1281.537314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1281.601952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1281.640953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1281.704335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1281.775696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1281.831289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.201.22@tcp [ 1281.871730] LNetError: 68636:0:(api-ni.c:4472:lnet_ni_set_healthv()) manually adding local NI 192.168.201.22@tcp to recovery [ 1282.336628] LNet: There was an unexpected network error while writing to 192.168.201.22: rc = -22 [ 1285.616169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1288.484887] LNet: There was an unexpected network error while writing to 192.168.201.22: rc = -22 [ 1288.497161] LNet: Skipped 1 previous similar message [ 1292.815478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1292.894365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1292.958636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1300.441720] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 02:26:44 (1786170404) [ 1300.656207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1300.676808] LNet: 69345:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1300.686612] LNetError: 69345:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1300.696378] LNet: Removed LNI 192.168.201.22@tcp [ 1300.704130] LNet: Removed LNI 192.168.201.22@tcp1 [ 1300.719770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1300.803243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1300.856778] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 1301.516909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1301.527635] LNet: Added LNI 192.168.201.22@tcp1 [8/256/0/180] [ 1302.135734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.22@tcp [ 1302.195677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1302.221468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1302.254702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1302.282810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1302.310051] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1302.359346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.201.22@tcp --health 0 [ 1302.518768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1302.816555] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.22@tcp [ 1303.840708] LNet: There was an unexpected network error while writing to 192.168.201.22: rc = -22 [ 1303.851469] LNet: Skipped 1 previous similar message [ 1318.248377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1318.453980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1318.504687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1318.552919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.201.22@tcp --health 500 [ 1318.660600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1324.438142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1324.510638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1324.539432] LNet: 69829:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1324.546508] LNetError: 69829:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1324.560173] LNetError: 69829:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1324.574182] LNet: Removed LNI 192.168.201.22@tcp [ 1324.593771] LNet: Removed LNI 192.168.201.22@tcp1 [ 1324.648478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1324.745494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1324.765431] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 1325.541871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1325.560595] LNet: Added LNI 192.168.201.22@tcp1 [8/256/0/180] [ 1326.481350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.22@tcp [ 1326.518139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1326.590690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1326.628280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1326.661657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1326.702986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1326.762680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1326.811712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.201.22@tcp --health 0 [ 1327.776918] LNet: There was an unexpected network error while writing to 192.168.201.22: rc = -22 [ 1327.783234] LNet: Skipped 3 previous similar messages [ 1328.218854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1338.669343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1338.708117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1338.739795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1345.825176] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 02:27:29 (1786170449) [ 1346.196025] LNet: 70876:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1346.202064] LNetError: 70876:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1346.209118] LNetError: 70876:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1346.214712] LNet: Removed LNI 192.168.201.22@tcp [ 1346.223319] LNet: Removed LNI 192.168.201.22@tcp1 [ 1347.011947] Key type .llcrypt unregistered [ 1347.014899] Key type ._llcrypt unregistered [ 1353.962272] Key type ._llcrypt registered [ 1353.966164] Key type .llcrypt registered [ 1354.152571] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1354.167829] alg: No test for adler32 (adler32-zlib) [ 1355.126331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1355.170757] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 1364.067036] Lustre: DEBUG MARKER: Initial discovery [ 1364.125567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.201.122@tcp [ 1366.582211] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1366.635364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1366.713435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.201.122@tcp [ 1366.776501] LNet: There was an unexpected network error while writing to 192.168.201.122: rc = -22 [ 1366.814357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1366.873291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.201.122@tcp -r 1 -m GET -l 3 [ 1366.950541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.201.122@tcp -r 1 -m GET -e local_error [ 1366.975533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.201.122@tcp -r 1 -m PUT -l 6 [ 1367.456381] LNet: 1 local NIs in recovery (showing 1): 192.168.201.22@tcp [ 1368.351789] Lustre: DEBUG MARKER: Force oleg122-client.virtnet to discover 192.168.201.122@tcp (in background) [ 1368.600643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.201.122@tcp [ 1370.075439] Lustre: DEBUG MARKER: Force oleg122-server to discover 192.168.201.22@tcp [ 1371.681317] LNet: There was an unexpected network error while writing to 192.168.201.122: rc = -22 [ 1372.222273] Lustre: DEBUG MARKER: Wait for 71643 [ 1373.696557] Lustre: DEBUG MARKER: Finished wait on 71643 [ 1373.847702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay del -a [ 1373.880282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1373.882149] LNet: There was an unexpected network error while writing to 192.168.201.122: rc = -22 [ 1374.212276] LNet: 72095:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1374.217328] LNetError: 72095:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1374.228789] LNet: Removed LNI 192.168.201.22@tcp [ 1374.586460] Key type .llcrypt unregistered [ 1374.588915] Key type ._llcrypt unregistered [ 1392.346258] Key type ._llcrypt registered [ 1392.347963] Key type .llcrypt registered [ 1393.817432] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 02:28:17 (1786170497) [ 1394.470748] Key type .llcrypt unregistered [ 1394.474134] Key type ._llcrypt unregistered [ 1401.053419] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1401.245765] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1401.249667] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1401.254277] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1403.898426] Key type ._llcrypt registered [ 1403.903971] Key type .llcrypt registered [ 1404.151186] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1404.163628] alg: No test for adler32 (adler32-zlib) [ 1405.065490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1405.117585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1405.142753] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 1405.761058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1405.780610] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1412.827395] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 02:28:36 (1786170516) [ 1412.891195] LNet: 74171:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1412.899946] LNetError: 74171:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1412.907457] LNet: Removed LNI 10.1.2.3@tcp [ 1412.923226] LNet: Removed LNI 192.168.201.22@tcp [ 1413.638673] Key type .llcrypt unregistered [ 1413.641272] Key type ._llcrypt unregistered [ 1420.156716] Key type ._llcrypt registered [ 1420.162113] Key type .llcrypt registered [ 1420.473902] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1420.497791] alg: No test for adler32 (adler32-zlib) [ 1421.519498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1421.583757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1421.609573] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 1422.532751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1422.564662] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1423.920762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.201.22@tcp [ 1423.976764] LNetError: Unexpected error -22 connecting to 192.168.201.22@tcp at host 192.168.201.22:988 [ 1430.760183] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 02:28:54 (1786170534) [ 1431.078609] LNet: 75318:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1431.090836] LNetError: 75318:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1431.098676] LNet: Removed LNI 10.1.2.3@tcp [ 1431.107640] LNet: Removed LNI 192.168.201.22@tcp [ 1431.755409] Key type .llcrypt unregistered [ 1431.757188] Key type ._llcrypt unregistered [ 1438.723165] Key type ._llcrypt registered [ 1438.733765] Key type .llcrypt registered [ 1439.012729] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1439.030946] alg: No test for adler32 (adler32-zlib) [ 1439.993764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1440.059496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1440.099953] LNet: Added LNI 192.168.201.22@tcp1 [8/256/0/180] [ 1440.889652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1440.928030] LNet: Added LNI 192.168.201.22@tcp2 [8/256/0/180] [ 1441.955953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.201.22@tcp1 --nid 192.168.201.22@tcp2 [ 1445.900987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.22@tcp1 192.168.201.22@tcp2 [ 1445.923584] LNetError: 76157:0:(lib-move.c:2101:lnet_handle_find_routed_path()) Unable to send message from 192.168.201.22@tcp1 to 192.168.201.22@tcp2 - Route table may be misconfigured [ 1445.987300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.22@tcp2 192.168.201.22@tcp1 [ 1452.677136] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 02:29:16 (1786170556) [ 1452.797987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1452.825353] LNet: 76726:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1452.841125] LNetError: 76726:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1452.851799] LNet: Removed LNI 192.168.201.22@tcp1 [ 1453.928598] LNet: Removed LNI 192.168.201.22@tcp2 [ 1453.989654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1454.100988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1454.166597] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 1455.281257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1455.300326] LNet: Added LNI 192.168.201.22@tcp1 [8/256/0/180] [ 1456.448772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.22@tcp [ 1456.519556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.22@tcp 192.168.201.22@tcp [ 1456.596179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.22@tcp1 192.168.201.22@tcp1 [ 1456.634861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.201.22@tcp -d 192.168.201.22@tcp -e network_timeout [ 1456.667286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.201.22@tcp -d 192.168.201.22@tcp1 -e network_timeout [ 1456.694660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.201.22@tcp1 -d 192.168.201.22@tcp -e network_timeout [ 1456.732592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.201.22@tcp1 -d 192.168.201.22@tcp1 -e network_timeout [ 1456.788247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.22@tcp [ 1456.819609] LNet: There was an unexpected network error while writing to 192.168.201.22: rc = -22 [ 1456.836951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1456.934914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1457.063731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1457.124630] LNet: 1 local NIs in recovery (showing 1): 192.168.201.22@tcp [ 1463.647834] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 02:29:27 (1786170567) [ 1463.730372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1463.761012] LNet: 77586:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1463.770795] LNetError: 77586:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1463.779124] LNetError: 77586:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1463.795464] LNet: Removed LNI 192.168.201.22@tcp [ 1464.813967] LNet: Removed LNI 192.168.201.22@tcp1 [ 1464.853938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1464.962726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1465.318860] LNet: 77633:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1465.704492] Key type .llcrypt unregistered [ 1465.708642] Key type ._llcrypt unregistered [ 1477.631491] Key type ._llcrypt registered [ 1477.633878] Key type .llcrypt registered [ 1479.046597] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 02:29:42 (1786170582) [ 1479.498182] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1479.510301] alg: No test for adler32 (adler32-zlib) [ 1480.525589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1481.934936] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1483.591193] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 02:29:47 (1786170587) [ 1483.956697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1484.005816] LNet: 78732:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1484.048393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1484.130604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1484.179725] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 1485.125963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1485.164462] LNet: Added LNI 192.168.201.22@tcp1 [8/256/0/180] [ 1486.161704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.22@tcp [ 1486.190608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.22@tcp1 [ 1486.263284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.22@tcp1 [ 1493.589228] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 02:29:56 (1786170596) [ 1495.294984] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg122-server [ 1497.782870] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 02:30:01 (1786170601) [ 1499.738626] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg122-server [ 1501.358597] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 02:30:05 (1786170605) [ 1503.040094] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg122-server [ 1504.391437] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 02:30:08 (1786170608) [ 1507.340333] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg122-server [ 1509.418895] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 02:30:13 (1786170613) [ 1511.406915] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg122-server [ 1512.996829] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 02:30:16 (1786170616) [ 1514.785155] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg122-server [ 1516.371104] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 02:30:20 (1786170620) [ 1517.975726] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg122-server [ 1519.349407] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 02:30:23 (1786170623) [ 1521.064856] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg122-server [ 1522.949869] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 02:30:26 (1786170626) [ 1525.117749] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg122-server [ 1526.739479] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 02:30:30 (1786170630) [ 1526.983648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1527.014534] LNet: 82611:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1527.029803] LNetError: 82611:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1527.037853] LNet: Removed LNI 192.168.201.22@tcp [ 1528.107094] LNet: Removed LNI 192.168.201.22@tcp1 [ 1528.177964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1528.284908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1528.341929] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 1529.317950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1529.401889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.22@tcp [ 1531.957975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1531.990351] LNet: 82776:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1531.997628] LNetError: 82776:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1532.003947] LNetError: 82776:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1532.015532] LNet: Removed LNI 192.168.201.22@tcp [ 1532.048304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1532.152691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1532.201621] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 1533.251633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.22@tcp [ 1534.688775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1534.719668] LNet: 82931:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1534.725830] LNetError: 82931:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1534.743418] LNet: Removed LNI 192.168.201.22@tcp [ 1534.774680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1534.829823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1534.856350] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 1536.048620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1536.163241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.22@tcp [ 1536.388419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1537.458041] LNet: Removed LNI 192.168.201.22@tcp [ 1537.505792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1537.617993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1537.664912] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 1538.858949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.22@tcp [ 1540.353322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1540.369560] LNet: 83233:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1540.375744] LNet: 83233:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1540.390162] LNetError: 83233:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1540.399402] LNetError: 83233:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1540.443057] LNet: Removed LNI 192.168.201.22@tcp [ 1540.499775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1540.647477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1540.719731] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 1541.724750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1541.868828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.22@tcp [ 1542.087398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1543.143860] LNet: Removed LNI 192.168.201.22@tcp [ 1543.209463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1543.380383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1543.428089] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 1544.348537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.22@tcp [ 1546.076818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1546.131967] LNetError: 83535:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1546.138806] LNetError: 83535:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1546.155586] LNet: Removed LNI 192.168.201.22@tcp [ 1546.199871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1546.328043] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1546.378271] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 1547.398296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1547.476916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.22@tcp [ 1547.679279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1547.727886] LNet: Removed LNI 192.168.201.22@tcp [ 1547.744661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1547.805494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1547.831258] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 1548.662388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.22@tcp [ 1548.972318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1548.994888] LNet: 83828:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1549.004249] LNet: 83828:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 1550.069800] LNet: Removed LNI 192.168.201.22@tcp [ 1550.107510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1550.210747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1550.239687] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 1550.979245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1551.045116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.22@tcp [ 1551.305603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1552.377031] LNet: Removed LNI 192.168.201.22@tcp [ 1552.434897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1552.558517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1552.595906] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 1553.527748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.22@tcp [ 1553.767859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1554.854262] LNet: Removed LNI 192.168.201.22@tcp [ 1554.897241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1555.002858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1555.026585] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 1555.916239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1556.011918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.22@tcp [ 1557.684831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1557.720468] LNetError: 84277:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1557.724436] LNetError: 84277:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 1558.762243] LNet: Removed LNI 192.168.201.22@tcp [ 1558.801948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1558.922500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1558.976563] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 1560.035725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.22@tcp [ 1560.419672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1561.524349] LNet: Removed LNI 192.168.201.22@tcp [ 1561.570926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1561.706549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1561.766140] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 1562.773556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1562.878938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1562.941190] LNet: Removed LNI 192.168.201.22@tcp [ 1562.975184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1563.107754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1563.144415] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 1564.275905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1564.354718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1564.385584] LNet: Removed LNI 192.168.201.22@tcp [ 1564.399146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1564.445564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 128 [ 1564.510652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1564.565313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1564.656689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 1000000000000000000000000 [ 1564.755237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1564.791230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1564.839720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer foo [ 1564.906210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1564.934837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1564.977329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 8 [ 1564.998575] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 1565.748620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1565.787103] LNet: 84863:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1565.795407] LNet: 84863:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 9 previous similar messages [ 1565.816920] LNet: Removed LNI 192.168.201.22@tcp [ 1565.842437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1565.892796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1572.980578] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 02:31:16 (1786170676) [ 1573.154095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1573.258898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1573.371957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1573.410213] LNet: Added LNI 192.168.201.22@tcp231 [8/256/0/180] [ 1574.721046] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1574.747528] LNetError: 85583:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1574.754836] LNetError: 85583:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 1574.766216] LNet: Removed LNI 192.168.201.22@tcp231 [ 1574.787947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1574.864766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1574.888285] LNet: Added LNI 192.168.201.22@tcp231 [8/256/0/180] [ 1574.898860] LNet: 85591:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1575.399142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1575.445453] LNet: Removed LNI 192.168.201.22@tcp231 [ 1575.458777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1575.493694] LNet: Added LNI 192.168.201.22@tcp231 [8/256/0/180] [ 1576.666235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1576.703337] LNet: Removed LNI 192.168.201.22@tcp231 [ 1576.737571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1576.801848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1576.840162] LNet: Added LNI 192.168.201.22@tcp231 [8/256/0/0] [ 1576.844941] LNet: 85743:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1577.410427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1577.454575] LNet: Removed LNI 192.168.201.22@tcp231 [ 1577.469962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1577.505731] LNet: Added LNI 192.168.201.22@tcp231 [8/256/0/0] [ 1578.782255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1578.822977] LNet: Removed LNI 192.168.201.22@tcp231 [ 1578.855127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1578.949364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1578.993644] LNet: Added LNI 192.168.201.22@tcp231 [8/256/0/60] [ 1579.006497] LNet: 85894:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1579.622802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1579.695526] LNet: Removed LNI 192.168.201.22@tcp231 [ 1579.731761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1579.769916] LNet: Added LNI 192.168.201.22@tcp231 [8/256/0/60] [ 1581.266207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1581.299326] LNet: Removed LNI 192.168.201.22@tcp231 [ 1581.311240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1581.367645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1581.408962] LNet: Added LNI 192.168.201.22@tcp231 [8/256/0/180] [ 1581.418810] LNet: 86046:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1581.778566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1581.799558] LNet: Removed LNI 192.168.201.22@tcp231 [ 1581.809605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1581.835579] LNet: Added LNI 192.168.201.22@tcp231 [8/256/0/180] [ 1589.880340] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 02:31:33 (1786170693) [ 1590.101377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1590.221608] LNet: Removed LNI 192.168.201.22@tcp231 [ 1590.251458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1590.304909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1590.322555] LNet: Added LNI 192.168.201.22@tcp232 [8/256/0/180] [ 1591.323383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1591.381845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1591.431207] LNet: Removed LNI 192.168.201.22@tcp232 [ 1591.456154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1591.499941] LNet: Added LNI 192.168.201.22@tcp232 [8/256/0/180] [ 1592.408245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1592.460146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1592.511253] LNet: Removed LNI 192.168.201.22@tcp232 [ 1592.530635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1592.586664] LNet: Added LNI 192.168.201.22@tcp232 [8/256/0/180] [ 1593.882812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1593.923691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1593.970175] LNet: Removed LNI 192.168.201.22@tcp232 [ 1594.015371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1594.066319] LNet: Added LNI 192.168.201.22@tcp232 [8/256/0/180] [ 1595.235857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1595.284213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1595.337655] LNet: Removed LNI 192.168.201.22@tcp232 [ 1595.379852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1602.793330] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 02:31:46 (1786170706) [ 1603.356393] LNet: 87886:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1603.377534] LNet: 87886:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 6 previous similar messages [ 1604.210366] Key type .llcrypt unregistered [ 1604.212912] Key type ._llcrypt unregistered [ 1611.367526] Key type ._llcrypt registered [ 1611.369567] Key type .llcrypt registered [ 1611.719957] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1611.786432] alg: No test for adler32 (adler32-zlib) [ 1612.699348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1612.727611] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 1620.792622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.122@tcp [ 1631.900565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1631.982616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1632.039649] LNet: Added LNI 192.168.201.22@tcp1 [8/256/0/180] [ 1633.916992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp -d 192.168.201.122@tcp -m GET -e local_interrupt -r 2 [ 1633.950860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp -d 192.168.201.122@tcp1 -m GET -e local_interrupt -r 2 [ 1633.980185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.22@tcp -d 192.168.201.22@tcp -m GET -e local_interrupt -r 2 [ 1634.041297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp -d 192.168.201.22@tcp1 -m GET -e local_interrupt -r 2 [ 1634.074874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp1 -d 192.168.201.122@tcp -m GET -e local_interrupt -r 2 [ 1634.106994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp1 -d 192.168.201.122@tcp1 -m GET -e local_interrupt -r 2 [ 1634.136845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.22@tcp1 -d 192.168.201.22@tcp -m GET -e local_interrupt -r 2 [ 1634.170974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.22@tcp1 -d 192.168.201.22@tcp1 -m GET -e local_interrupt -r 2 [ 1634.214551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.22@tcp 192.168.201.122@tcp [ 1634.234582] LNet: There was an unexpected network error while writing to 192.168.201.122: rc = -22 [ 1634.245475] LNet: 1 local NIs in recovery (showing 1): 192.168.201.22@tcp [ 1634.541705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1634.583279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1634.616758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1634.650527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1634.684724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1634.717964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1634.781547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1634.824928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1634.844714] LNet: 88554:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1634.865103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1634.905931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1634.932784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1634.961221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1634.997175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1635.031772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1635.649275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.22@tcp -d 192.168.201.122@tcp -m GET -e local_dropped -r 2 [ 1635.694685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp -d 192.168.201.122@tcp1 -m GET -e local_dropped -r 2 [ 1635.735522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.22@tcp -d 192.168.201.22@tcp -m GET -e local_dropped -r 2 [ 1635.783575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp -d 192.168.201.22@tcp1 -m GET -e local_dropped -r 2 [ 1635.821639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp1 -d 192.168.201.122@tcp -m GET -e local_dropped -r 2 [ 1635.856624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp1 -d 192.168.201.122@tcp1 -m GET -e local_dropped -r 2 [ 1635.897458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.22@tcp1 -d 192.168.201.22@tcp -m GET -e local_dropped -r 2 [ 1635.962178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.22@tcp1 -d 192.168.201.22@tcp1 -m GET -e local_dropped -r 2 [ 1636.024962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.22@tcp 192.168.201.122@tcp [ 1636.047968] LNet: There was an unexpected network error while writing to 192.168.201.122: rc = -22 [ 1636.431599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1636.454533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1636.486262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1636.508719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1636.533531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1636.570688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1636.600765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1636.633956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1636.652710] LNet: 88675:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1636.670860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1636.703353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1636.745052] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1636.794958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1636.865930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1636.912992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1637.564976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.22@tcp -d 192.168.201.122@tcp -m GET -e local_aborted -r 2 [ 1637.607826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp -d 192.168.201.122@tcp1 -m GET -e local_aborted -r 2 [ 1637.657766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.22@tcp -d 192.168.201.22@tcp -m GET -e local_aborted -r 2 [ 1637.701463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.22@tcp -d 192.168.201.22@tcp1 -m GET -e local_aborted -r 2 [ 1637.739736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp1 -d 192.168.201.122@tcp -m GET -e local_aborted -r 2 [ 1637.785266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp1 -d 192.168.201.122@tcp1 -m GET -e local_aborted -r 2 [ 1637.851965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.22@tcp1 -d 192.168.201.22@tcp -m GET -e local_aborted -r 2 [ 1637.947765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.22@tcp1 -d 192.168.201.22@tcp1 -m GET -e local_aborted -r 2 [ 1638.026238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.22@tcp 192.168.201.122@tcp [ 1638.049917] LNet: There was an unexpected network error while writing to 192.168.201.122: rc = -22 [ 1638.304255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1638.386551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1638.428550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1638.503239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1638.574955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1638.625578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1638.655303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1638.690033] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1638.709775] LNet: 88795:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1638.738230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1638.810458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1638.884324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1638.933610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1639.007635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1639.069679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1639.804142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp -d 192.168.201.122@tcp -m GET -e local_no_route -r 2 [ 1639.870478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp -d 192.168.201.122@tcp1 -m GET -e local_no_route -r 2 [ 1639.921534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.22@tcp -d 192.168.201.22@tcp -m GET -e local_no_route -r 2 [ 1639.956905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.22@tcp -d 192.168.201.22@tcp1 -m GET -e local_no_route -r 2 [ 1640.007668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.22@tcp1 -d 192.168.201.122@tcp -m GET -e local_no_route -r 2 [ 1640.097413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp1 -d 192.168.201.122@tcp1 -m GET -e local_no_route -r 2 [ 1640.160325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp1 -d 192.168.201.22@tcp -m GET -e local_no_route -r 2 [ 1640.221985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp1 -d 192.168.201.22@tcp1 -m GET -e local_no_route -r 2 [ 1640.295068] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.22@tcp 192.168.201.122@tcp [ 1640.334214] LNet: There was an unexpected network error while writing to 192.168.201.122: rc = -22 [ 1640.574784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1640.603855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1640.636799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1640.664578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1640.690803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1640.718784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1640.757475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1640.795385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1640.816345] LNet: 88915:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1640.840826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1640.893980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1640.929686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1640.974208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1641.015260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1641.045970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1641.577771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.22@tcp -d 192.168.201.122@tcp -m GET -e local_timeout -r 2 [ 1641.607414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.22@tcp -d 192.168.201.122@tcp1 -m GET -e local_timeout -r 2 [ 1641.639964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp -d 192.168.201.22@tcp -m GET -e local_timeout -r 2 [ 1641.677598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.22@tcp -d 192.168.201.22@tcp1 -m GET -e local_timeout -r 2 [ 1641.723728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.22@tcp1 -d 192.168.201.122@tcp -m GET -e local_timeout -r 2 [ 1641.766569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp1 -d 192.168.201.122@tcp1 -m GET -e local_timeout -r 2 [ 1641.815401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.22@tcp1 -d 192.168.201.22@tcp -m GET -e local_timeout -r 2 [ 1641.844525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp1 -d 192.168.201.22@tcp1 -m GET -e local_timeout -r 2 [ 1641.896272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.22@tcp 192.168.201.122@tcp [ 1642.199910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1642.237343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1642.280506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1642.325793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1642.378447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1642.421718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1642.473424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1642.534830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1642.593713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1642.655668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1642.714650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1642.754794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1642.799290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1642.850995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1643.418445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp -d 192.168.201.122@tcp -m GET -e remote_dropped -r 2 [ 1643.448709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.22@tcp -d 192.168.201.122@tcp1 -m GET -e remote_dropped -r 2 [ 1643.479519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.22@tcp -d 192.168.201.22@tcp -m GET -e remote_dropped -r 2 [ 1643.506414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp -d 192.168.201.22@tcp1 -m GET -e remote_dropped -r 2 [ 1643.533885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp1 -d 192.168.201.122@tcp -m GET -e remote_dropped -r 2 [ 1643.573375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp1 -d 192.168.201.122@tcp1 -m GET -e remote_dropped -r 2 [ 1643.639475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.22@tcp1 -d 192.168.201.22@tcp -m GET -e remote_dropped -r 2 [ 1643.686302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp1 -d 192.168.201.22@tcp1 -m GET -e remote_dropped -r 2 [ 1643.735329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.22@tcp 192.168.201.122@tcp [ 1643.768161] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.122@tcp [ 1644.004681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1644.041966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1644.082820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1644.122259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1644.165365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1644.210508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1644.254623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1644.318492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1644.371260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1644.431583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1644.478818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1644.528777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1644.573758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1644.645187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1649.811691] LNet: 89234:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1649.819727] LNetError: 89234:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1649.825783] LNet: Removed LNI 192.168.201.22@tcp [ 1649.834218] LNet: Removed LNI 192.168.201.22@tcp1 [ 1650.117691] Key type .llcrypt unregistered [ 1650.119747] Key type ._llcrypt unregistered [ 1667.373569] Key type ._llcrypt registered [ 1667.375608] Key type .llcrypt registered [ 1668.657239] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 02:32:52 (1786170772) [ 1669.269763] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1669.292269] alg: No test for adler32 (adler32-zlib) [ 1670.236581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1670.292659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1670.307774] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 1671.173286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1671.201624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.201.22@tcp [ 1671.216758] LNetError: 90213:0:(api-ni.c:4472:lnet_ni_set_healthv()) manually adding local NI 192.168.201.22@tcp to recovery [ 1671.303148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.201.22@tcp [ 1671.399638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1678.784862] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 02:33:02 (1786170782) [ 1679.098861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1679.132749] LNet: 90804:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1679.145572] LNetError: 90804:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1679.164524] LNet: Removed LNI 192.168.201.22@tcp [ 1679.186044] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1679.254940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1679.287395] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 1680.436170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1680.483097] LNet: Added LNI 192.168.201.22@tcp1 [8/256/0/180] [ 1681.990917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.201.22@tcp [ 1682.021299] LNetError: 91070:0:(api-ni.c:4472:lnet_ni_set_healthv()) manually adding local NI 192.168.201.22@tcp to recovery [ 1682.272459] LNet: 1 local NIs in recovery (showing 1): 192.168.201.22@tcp [ 1682.273325] LNet: There was an unexpected network error while writing to 192.168.201.22: rc = -22 [ 1683.283144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.201.22@tcp [ 1691.931352] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 02:33:15 (1786170795) [ 1693.838564] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg122-server [ 1695.281120] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 02:33:19 (1786170799) [ 1697.603645] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg122-server [ 1699.469645] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 02:33:23 (1786170803) [ 1701.364974] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg122-server [ 1703.433455] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 02:33:27 (1786170807) [ 1703.581652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1703.605165] LNet: 92690:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1703.612287] LNetError: 92690:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1703.617507] LNet: Removed LNI 192.168.201.22@tcp [ 1703.635976] LNet: Removed LNI 192.168.201.22@tcp1 [ 1703.664428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1703.711376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1703.730418] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 1704.563711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1704.670494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1704.731935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1704.816788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1704.868779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1704.929894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1712.054319] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 02:33:35 (1786170815) [ 1712.378071] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1712.397284] LNet: 93417:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1712.403986] LNetError: 93417:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1712.408631] LNetError: 93417:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1712.435549] LNet: Removed LNI 192.168.201.22@tcp [ 1712.450767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1712.487207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1712.503573] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 1719.116791] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 02:33:43 (1786170823) [ 1719.382121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1719.411933] LNet: 94121:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1719.418837] LNetError: 94121:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1719.432974] LNet: Removed LNI 192.168.201.22@tcp [ 1719.473694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1719.584202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1719.617254] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 1727.389758] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 02:33:51 (1786170831) [ 1729.299437] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 1730.992127] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 02:33:54 (1786170834) [ 1731.634653] LNet: 95156:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1731.640249] LNetError: 95156:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1731.657269] LNet: Removed LNI 192.168.201.22@tcp [ 1732.280616] Key type .llcrypt unregistered [ 1732.282646] Key type ._llcrypt unregistered [ 1739.403881] Key type ._llcrypt registered [ 1739.408884] Key type .llcrypt registered [ 1739.599151] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1739.614927] alg: No test for adler32 (adler32-zlib) [ 1740.465825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1740.500657] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 1748.103666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.122@tcp [ 1755.023506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.201.122@tcp [ 1755.445865] LNet: 95583:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1755.452344] LNetError: 95583:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1755.467935] LNet: Removed LNI 192.168.201.22@tcp [ 1755.951690] Key type .llcrypt unregistered [ 1755.954097] Key type ._llcrypt unregistered [ 1768.028455] Key type ._llcrypt registered [ 1768.030026] Key type .llcrypt registered [ 1769.217719] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 1770.775942] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 1772.442918] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 02:34:36 (1786170876) [ 1773.873422] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 [ 1775.419777] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 02:34:39 (1786170879) [ 1777.372395] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg122-server [ 1779.313576] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 02:34:43 (1786170883) [ 1781.020744] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg122-server [ 1782.632541] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 02:34:46 (1786170886) [ 1784.135626] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1784.228157] alg: No test for adler32 (adler32-zlib) [ 1785.411845] Lustre: Lustre: Build Version: 2.17.56_50_gc905d6b [ 1785.809741] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 1787.504346] Key type lgssc registered [ 1789.078544] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1809.159342] Key type lgssc unregistered [ 1809.468951] LNet: 98761:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1809.479792] LNetError: 98761:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1809.491125] LNet: Removed LNI 192.168.201.22@tcp [ 1810.172634] Key type .llcrypt unregistered [ 1810.175872] Key type ._llcrypt unregistered [ 1824.641204] Key type ._llcrypt registered [ 1824.643150] Key type .llcrypt registered [ 1826.192751] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 02:35:29 (1786170929) [ 1826.680587] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1826.693994] alg: No test for adler32 (adler32-zlib) [ 1827.685741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1827.786221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1827.837978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1827.903626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1827.945978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1834.126192] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 02:35:38 (1786170938) [ 1834.436304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1834.459233] LNet: 100178:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1834.483413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1834.532502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1834.564565] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 1835.479994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1835.497556] LNet: 100322:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1835.503339] LNetError: 100322:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1835.510602] LNet: Removed LNI 192.168.201.22@tcp [ 1835.522599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1835.571521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1835.606975] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 1836.285771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1836.330290] LNetError: 100350:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1836.346449] LNet: Removed LNI 192.168.201.22@tcp [ 1836.370965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1836.420876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1836.446414] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 1842.269708] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 02:35:46 (1786170946) [ 1842.470965] LNet: 100941:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1842.476217] LNet: 100941:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1842.481205] LNetError: 100941:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1842.491393] LNet: Removed LNI 192.168.201.22@tcp [ 1843.052726] Key type .llcrypt unregistered [ 1843.054449] Key type ._llcrypt unregistered [ 1849.548245] Key type ._llcrypt registered [ 1849.550434] Key type .llcrypt registered [ 1849.695770] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1849.709383] alg: No test for adler32 (adler32-zlib) [ 1850.610578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1850.672507] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 1850.674696] LNet: *** cfs_fail_loc=e003, val=100*** [ 1851.179147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.22@tcp [ 1851.191574] LNet: *** cfs_fail_loc=e003, val=100*** [ 1851.199747] LNet: Skipped 2 previous similar messages [ 1851.209472] Ping reply received with mapping: [ 1851.211168] found metadata reply NI[192.168.201.22@tcp] [ 1851.213587] 16 c9 a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 1851.216452] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 1851.219241] 5a 5a 5a 5a 5a 5a 5a 5a [ 1851.357620] LNet: 101342:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1851.375919] LNetError: 101342:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1851.393141] LNet: Removed LNI 192.168.201.22@tcp [ 1852.254391] Key type .llcrypt unregistered [ 1852.256859] Key type ._llcrypt unregistered [ 1864.381946] Key type ._llcrypt registered [ 1864.385603] Key type .llcrypt registered [ 1866.086910] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 02:36:09 (1786170969) [ 1866.694912] Key type .llcrypt unregistered [ 1866.697711] Key type ._llcrypt unregistered [ 1875.079389] Key type ._llcrypt registered [ 1875.083430] Key type .llcrypt registered [ 1875.528425] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1875.553405] alg: No test for adler32 (adler32-zlib) [ 1876.839417] LNetError: 102401:0:(api-ni.c:2849:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 1877.061743] Key type .llcrypt unregistered [ 1877.063525] Key type ._llcrypt unregistered [ 1882.035366] Key type ._llcrypt registered [ 1882.038835] Key type .llcrypt registered [ 1883.307102] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 02:36:27 (1786170987) [ 1883.874430] Key type .llcrypt unregistered [ 1883.876978] Key type ._llcrypt unregistered [ 1890.808408] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1890.815808] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1893.541734] Key type ._llcrypt registered [ 1893.544177] Key type .llcrypt registered [ 1893.710387] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1893.724531] alg: No test for adler32 (adler32-zlib) [ 1894.723154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1894.839343] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1904.837091] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 02:36:48 (1786171008) [ 1904.922886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1904.960510] LNet: 104181:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1904.967913] LNetError: 104181:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1904.993642] LNet: Removed LNI 10.1.2.3@tcp [ 1905.010391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1905.825995] Key type .llcrypt unregistered [ 1905.834537] Key type ._llcrypt unregistered [ 1913.279257] Key type ._llcrypt registered [ 1913.282388] Key type .llcrypt registered [ 1913.425669] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1913.445193] alg: No test for adler32 (adler32-zlib) [ 1914.468409] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 1914.551337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.22@tcp [ 1914.773919] LNet: 104482:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1914.785447] LNetError: 104482:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1914.818420] LNet: Removed LNI 192.168.201.22@tcp [ 1915.421327] Key type .llcrypt unregistered [ 1915.426914] Key type ._llcrypt unregistered [ 1926.370759] Key type ._llcrypt registered [ 1926.373450] Key type .llcrypt registered [ 1927.637413] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 02:37:11 (1786171031) [ 1928.229892] Key type .llcrypt unregistered [ 1928.233830] Key type ._llcrypt unregistered [ 1934.552789] Key type ._llcrypt registered [ 1934.555142] Key type .llcrypt registered [ 1934.640061] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1934.648866] alg: No test for adler32 (adler32-zlib) [ 1941.999772] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 02:37:25 (1786171045) [ 1942.139363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1942.173315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1942.251346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1942.272605] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 1943.283347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1943.337360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 1943.385898] LNet: Removed LNI 192.168.201.22@tcp [ 1943.675485] LNet: 106241:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1944.185819] Key type .llcrypt unregistered [ 1944.192904] Key type ._llcrypt unregistered [ 1955.946932] Key type ._llcrypt registered [ 1955.949874] Key type .llcrypt registered [ 1957.535470] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 02:37:41 (1786171061) [ 1958.006549] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1958.027485] alg: No test for adler32 (adler32-zlib) [ 1958.972238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1959.037932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1959.093737] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 1960.114533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.22@tcp [ 1960.160641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.201.22@tcp [ 1960.183908] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 1960.188314] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 1960.192242] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1960, 0, 0 [ 1960.198255] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 1960.201661] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 1960.206217] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 1960.209927] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1960, 0, 0 [ 1960.213977] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 1960.217490] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1960.222103] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 1960.226253] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1960, 0, 2010 [ 1960.232491] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 1960.235705] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1960.239480] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 1960.242993] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1960, 0, 2010 [ 1960.252881] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 1968.182156] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 02:37:51 (1786171071) [ 1968.538782] LNet: 107781:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1968.542902] LNetError: 107781:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1969.579558] LNet: Removed LNI 192.168.201.22@tcp [ 1970.373467] Key type .llcrypt unregistered [ 1970.375291] Key type ._llcrypt unregistered [ 1976.522500] Key type ._llcrypt registered [ 1976.524689] Key type .llcrypt registered [ 1976.748140] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1976.759320] alg: No test for adler32 (adler32-zlib) [ 1977.691126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1977.713245] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 1984.677001] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.122@tcp [ 1993.207284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1993.274655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1993.324903] LNet: Added LNI 192.168.201.22@tcp1 [8/256/0/180] [ 1994.273067] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1994.518548] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1994.527290] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1994.533688] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1996.933811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 1996.961693] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 2007.838541] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 02:38:31 (1786171111) [ 2007.979861] LNet: 109285:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2007.986371] LNetError: 109285:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2007.997108] LNet: Removed LNI 192.168.201.22@tcp [ 2008.002443] LNet: Removed LNI 192.168.201.22@tcp1 [ 2008.030767] LNet: Removed LNI 10.1.2.3@tcp99 [ 2008.712921] Key type .llcrypt unregistered [ 2008.714741] Key type ._llcrypt unregistered [ 2017.259504] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2017.264559] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2019.953959] Key type ._llcrypt registered [ 2019.956757] Key type .llcrypt registered [ 2020.266747] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2020.308820] alg: No test for adler32 (adler32-zlib) [ 2021.263423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2021.320817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2021.368825] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 2022.303853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2022.325396] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2024.590891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 2024.620730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2024.651161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 2024.666820] LNetError: 110041:0:(peer.c:2362:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 2024.699572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2024.730745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.22@tcp [ 2025.145571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2025.232367] LNet: 110061:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2025.261368] LNetError: 110061:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2025.269744] LNet: Removed LNI 10.1.2.3@tcp [ 2026.345872] LNet: Removed LNI 192.168.201.22@tcp [ 2027.125636] Key type .llcrypt unregistered [ 2027.138654] Key type ._llcrypt unregistered [ 2038.939964] Key type ._llcrypt registered [ 2038.941836] Key type .llcrypt registered [ 2040.417451] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 02:39:04 (1786171144) [ 2040.765642] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2040.778762] alg: No test for adler32 (adler32-zlib) [ 2041.760775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2041.821624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2041.841628] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 2042.608518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.22@tcp [ 2042.689603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg122-client@tcp [ 2049.086917] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 02:39:12 (1786171152) [ 2049.210413] LNet: 111591:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2049.216294] LNetError: 111591:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2049.233113] LNet: Removed LNI 192.168.201.22@tcp [ 2049.895698] Key type .llcrypt unregistered [ 2049.897918] Key type ._llcrypt unregistered [ 2055.430752] Key type ._llcrypt registered [ 2055.433332] Key type .llcrypt registered [ 2055.652580] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2055.662501] alg: No test for adler32 (adler32-zlib) [ 2056.629437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 2056.665263] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 2056.706112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 2062.740704] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 02:39:26 (1786171166) [ 2064.338597] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 2066.251048] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 02:39:30 (1786171170) [ 2067.638770] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 2069.268492] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 02:39:33 (1786171173) [ 2069.368671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2069.409942] LNet: 113014:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2069.417022] LNetError: 113014:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2069.447849] LNet: Removed LNI 192.168.201.22@tcp [ 2069.482251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2069.553408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2069.573271] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 2070.168474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2070.193221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2076.112699] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 02:39:40 (1786171180) [ 2076.377754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2076.412224] LNet: 113731:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2076.422542] LNetError: 113731:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2076.440229] LNet: Removed LNI 192.168.201.22@tcp [ 2076.464713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2076.519818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 2076.560976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2077.146181] Key type .llcrypt unregistered [ 2077.150809] Key type ._llcrypt unregistered [ 2086.949961] Key type ._llcrypt registered [ 2086.952447] Key type .llcrypt registered [ 2088.113208] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 02:39:52 (1786171192) [ 2088.705178] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2088.723279] alg: No test for adler32 (adler32-zlib) [ 2089.661363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2089.724385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2089.746437] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 2090.367326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2090.421342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.22@tcp [ 2096.818416] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 02:40:00 (1786171200) [ 2096.924761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2096.957416] LNet: 115287:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2096.963742] LNetError: 115287:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2096.991415] LNet: Removed LNI 192.168.201.22@tcp [ 2097.050142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2097.097596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 2097.123914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 2103.825376] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 02:40:07 (1786171207) [ 2103.958441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2104.009308] LNet: 115865:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2104.048847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2104.188368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2104.545238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2104.578909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2104.641159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 2104.943635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2104.980596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2105.109679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2105.145370] LNet: 115904:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2105.170655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2105.251092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2105.724725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2105.762820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2105.876903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 2106.282734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2106.302977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2112.686240] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 02:40:16 (1786171216) [ 2112.800843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2112.836719] LNet: 116498:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2112.864614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2112.948074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2112.979528] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 2113.651224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2113.673427] LNet: Added LNI 192.168.201.22@tcp2 [8/256/0/180] [ 2114.446373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2114.512942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.22@tcp [ 2114.837909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2114.880234] LNetError: 116784:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2114.886949] LNet: Removed LNI 192.168.201.22@tcp [ 2114.905115] LNet: Removed LNI 192.168.201.22@tcp2 [ 2114.951666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2115.033696] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 2115.048139] LNet: Added LNI 192.168.201.22@tcp2 [8/256/0/180] [ 2115.059590] LNet: 116790:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2115.860290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.22@tcp [ 2116.164923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2116.193826] LNetError: 116827:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2116.199924] LNetError: 116827:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2116.207976] LNet: Removed LNI 192.168.201.22@tcp [ 2116.222892] LNet: Removed LNI 192.168.201.22@tcp2 [ 2116.263468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2116.324884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2116.346074] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 2117.120517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2117.149806] LNet: Added LNI 192.168.201.22@tcp2 [8/256/0/180] [ 2118.108339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.22@tcp [ 2118.156709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2118.204825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.22@tcp [ 2118.439230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2118.460308] LNet: 117112:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2118.466186] LNet: 117112:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 2118.471734] LNetError: 117112:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2118.478441] LNetError: 117112:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2118.484412] LNet: Removed LNI 192.168.201.22@tcp [ 2119.539988] LNet: Removed LNI 192.168.201.22@tcp2 [ 2119.621632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2119.791459] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 2119.801214] LNet: Added LNI 192.168.201.22@tcp2 [8/256/0/180] [ 2119.810878] LNet: 117118:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2120.884538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.22@tcp [ 2121.288513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2121.311536] LNet: Removed LNI 192.168.201.22@tcp [ 2122.349089] LNet: Removed LNI 192.168.201.22@tcp2 [ 2122.447470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2122.511676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2122.531088] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 2124.123674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2124.158417] LNet: Added LNI 192.168.201.22@tcp2 [8/256/0/180] [ 2125.492766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2125.550697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.22@tcp [ 2125.882785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2125.916577] LNetError: 117435:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2125.920548] LNetError: 117435:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 2125.947775] LNet: Removed LNI 192.168.201.22@tcp [ 2125.973864] LNet: Removed LNI 192.168.201.22@tcp2 [ 2126.007884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2126.099335] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 2126.110239] LNet: Added LNI 192.168.201.22@tcp2 [8/256/0/180] [ 2126.119240] LNet: 117441:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2126.555548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.22@tcp [ 2126.721862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2126.737434] LNet: 117478:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2126.741554] LNet: 117478:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 2127.776178] LNet: Removed LNI 192.168.201.22@tcp [ 2127.787115] LNet: Removed LNI 192.168.201.22@tcp2 [ 2127.819451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2127.893097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2127.928143] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 2128.680468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2128.704368] LNet: Added LNI 192.168.201.22@tcp2 [8/256/0/180] [ 2129.527714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.22@tcp [ 2129.565351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2129.610182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.22@tcp [ 2129.799949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2130.848179] LNet: Removed LNI 192.168.201.22@tcp [ 2131.888596] LNet: Removed LNI 192.168.201.22@tcp2 [ 2132.036876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2132.169216] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 2132.187966] LNet: Added LNI 192.168.201.22@tcp2 [8/256/0/180] [ 2132.200671] LNet: 117769:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2133.075355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.22@tcp [ 2133.310820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2133.330910] LNet: Removed LNI 192.168.201.22@tcp [ 2134.374912] LNet: Removed LNI 192.168.201.22@tcp2 [ 2134.424313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2134.520768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2134.556639] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 2135.479315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2135.497038] LNet: Added LNI 192.168.201.22@tcp2 [8/256/0/180] [ 2136.644384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2136.686957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2136.769581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.22@tcp [ 2137.250765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2137.276500] LNetError: 118103:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2137.284624] LNetError: 118103:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 2137.293052] LNet: Removed LNI 192.168.201.22@tcp [ 2137.305936] LNet: Removed LNI 192.168.201.22@tcp2 [ 2137.341967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2137.393515] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 2137.402355] LNet: Added LNI 192.168.201.22@tcp2 [8/256/0/180] [ 2137.408239] LNet: 118109:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2138.374033] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.22@tcp [ 2138.747534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2139.808176] LNet: Removed LNI 192.168.201.22@tcp [ 2139.821304] LNet: Removed LNI 192.168.201.22@tcp2 [ 2139.868966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2139.951090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2139.998231] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 2140.981527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2141.008452] LNet: Added LNI 192.168.201.22@tcp2 [8/256/0/180] [ 2142.177756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.22@tcp [ 2142.231505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2142.289646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2142.376688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.22@tcp [ 2142.908071] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2142.953286] LNet: 118463:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2142.965947] LNet: 118463:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 2144.032381] LNet: Removed LNI 192.168.201.22@tcp [ 2145.063908] LNet: Removed LNI 192.168.201.22@tcp2 [ 2145.129225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2145.252089] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 2145.263787] LNet: Added LNI 192.168.201.22@tcp2 [8/256/0/180] [ 2146.123411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.22@tcp [ 2152.902555] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 2154.348785] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 02:40:58 (1786171258) [ 2154.672142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2154.691985] LNetError: 119231:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2154.699410] LNetError: 119231:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 2154.705696] LNet: Removed LNI 192.168.201.22@tcp [ 2155.759770] LNet: Removed LNI 192.168.201.22@tcp2 [ 2155.812402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2156.175993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2156.199075] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 2156.734061] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2156.778075] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2157.643776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2157.661793] LNet: Removed LNI 10.1.2.3@tcp [ 2157.669785] LNet: Removed LNI 192.168.201.22@tcp [ 2157.682506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2157.718798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 2157.743522] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 2157.748492] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2158.221441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 2158.262033] LNet: Removed LNI 192.168.201.22@tcp [ 2158.274279] LNet: Removed LNI 10.1.2.3@tcp [ 2165.079737] Lustre: DEBUG MARKER: == sanity-lnet test 475: per-operation latency stats populate and reset ========================================================== 02:41:09 (1786171269) [ 2165.263993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2165.314729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2165.377292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2165.398922] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 2166.074300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2166.104589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.22@tcp [ 2166.143310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.22@tcp [ 2166.179593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.22@tcp [ 2166.208528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.22@tcp [ 2166.253527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.22@tcp [ 2166.281662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.22@tcp [ 2166.307758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.22@tcp [ 2166.333529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.22@tcp [ 2166.385853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.22@tcp [ 2166.414656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.22@tcp [ 2166.542973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2173.235612] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 02:41:17 (1786171277) [ 2173.420789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2174.503561] LNet: Removed LNI 192.168.201.22@tcp [ 2174.559599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2174.720880] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2174.863062] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2174.877630] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2174.887617] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2177.224691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2177.255645] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 2178.109360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2178.133585] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2180.259072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.22@tcp [ 2180.331966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 2180.411705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.201.22@tcp -m PUT -e local_timeout -r 1 [ 2180.485610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 2180.549827] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2180.554156] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2180.788171] LNetError: 120910:0:(lib-move.c:4801:lnet_parse()) 192.168.201.22@tcp, src 192.168.201.22@tcp: Dropping PUT (error -108 looking up sender) [ 2181.792993] LNet: 121372:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2181.805553] LNet: 121372:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 2181.812511] LNet: Removed LNI 10.1.2.3@tcp [ 2182.891643] LNet: Removed LNI 192.168.201.22@tcp [ 2183.365901] Key type .llcrypt unregistered [ 2183.368331] Key type ._llcrypt unregistered [ 2194.023233] Key type ._llcrypt registered [ 2194.025812] Key type .llcrypt registered [ 2195.148334] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 02:41:39 (1786171299) [ 2195.630592] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2195.647299] alg: No test for adler32 (adler32-zlib) [ 2196.568761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2196.901725] LNet: 122224:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2197.580739] Key type .llcrypt unregistered [ 2197.587726] Key type ._llcrypt unregistered [ 2203.889816] Key type ._llcrypt registered [ 2203.895486] Key type .llcrypt registered [ 2204.192603] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2204.214547] alg: No test for adler32 (adler32-zlib) [ 2205.125580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2205.148121] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 2212.329792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.122@tcp [ 2212.521163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 2212.970848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.201.122@tcp [ 2213.183816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2213.215647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2213.246403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2213.271452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2213.299194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2213.318734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2213.343083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2213.344365] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.122@tcp [ 2213.385699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2213.403471] LNet: 122676:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2213.423065] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2213.445616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2213.467814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2213.494775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2213.526887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2213.548659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2214.052530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2214.253436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2214.287900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2214.325701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2214.365446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2214.391587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2214.415028] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2214.447658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2214.473314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2214.494811] LNet: 122794:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2214.516833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2214.551914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2214.569185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2214.605507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2214.650938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2214.699715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2215.395447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.201.122@tcp [ 2215.679894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2215.706943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2215.726558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2215.752237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2215.773721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2215.804701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2215.837792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2215.870659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2215.890481] LNet: 122911:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2215.913525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2215.949879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2215.982067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2216.020485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2216.066946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2216.106121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2216.793684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2217.001407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2217.017168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2217.036808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2217.055660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2217.079178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2217.103211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2217.148750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2217.185385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2217.210451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2217.236807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2217.266658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2217.315901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2217.341435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2217.372531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2217.724962] LNet: 123078:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2217.731961] LNetError: 123078:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2217.747889] LNet: Removed LNI 192.168.201.22@tcp [ 2218.108938] Key type .llcrypt unregistered [ 2218.113757] Key type ._llcrypt unregistered [ 2234.493551] Key type ._llcrypt registered [ 2234.497030] Key type .llcrypt registered [ 2235.537710] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 02:42:19 (1786171339) [ 2235.978208] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2235.992911] alg: No test for adler32 (adler32-zlib) [ 2236.934671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2237.331229] LNet: 123966:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2238.018389] Key type .llcrypt unregistered [ 2238.020758] Key type ._llcrypt unregistered [ 2244.768217] Key type ._llcrypt registered [ 2244.769598] Key type .llcrypt registered [ 2244.997508] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2245.013753] alg: No test for adler32 (adler32-zlib) [ 2245.932695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2245.958380] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 2253.434639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.122@tcp [ 2261.418639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2261.480348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2261.527685] LNet: Added LNI 192.168.201.22@tcp1 [8/256/0/180] [ 2262.501193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 2263.374963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.201.122@tcp [ 2263.693770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2263.733860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2263.761869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2263.785121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2263.814134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2263.833268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2263.855955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2263.896507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2263.921672] LNet: 124622:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2263.942167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2263.991481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2264.047678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2264.071504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2264.117166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2264.143571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2264.616329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2264.808812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2264.830662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2264.852485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2264.872931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2264.901232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2264.925371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2264.973721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2265.030564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2265.055324] LNet: 124739:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2265.077777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2265.107951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2265.135552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2265.171652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2265.199565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2265.229756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2265.847930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.201.122@tcp [ 2266.106419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2266.132655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2266.154602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2266.178507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2266.210422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2266.235678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2266.264331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2266.291953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2266.308267] LNet: 124856:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2266.327868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2266.352602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2266.371482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2266.385752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2266.459542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2266.465672] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.122@tcp [ 2266.518686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2267.203425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2267.564137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2267.611053] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2267.646420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2267.709888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2267.767686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2267.820342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2267.851988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2267.883902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2267.920919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2267.953817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2267.990916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2268.022617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2268.048593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2268.083340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2274.040330] LNet: 125058:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2274.058341] LNetError: 125058:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2274.076310] LNet: Removed LNI 192.168.201.22@tcp [ 2274.089474] LNet: Removed LNI 192.168.201.22@tcp1 [ 2274.633849] Key type .llcrypt unregistered [ 2274.638553] Key type ._llcrypt unregistered [ 2295.410973] Key type ._llcrypt registered [ 2295.414657] Key type .llcrypt registered [ 2296.920604] Lustre: DEBUG MARKER: == sanity-lnet test 525: keep reachable peer on local net delete ========================================================== 02:43:20 (1786171400) [ 2297.316836] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2297.324666] alg: No test for adler32 (adler32-zlib) [ 2298.229539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2298.287642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2298.313916] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 2298.957891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2298.986523] LNet: Added LNI 192.168.201.22@tcp2 [8/256/0/180] [ 2299.728361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.22@tcp [ 2299.773890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp [ 2300.832388] LNet: Removed LNI 192.168.201.22@tcp [ 2307.324181] Lustre: DEBUG MARKER: == sanity-lnet test 550: DD-off: keep route up when gw reachable via another local NI ========================================================== 02:43:31 (1786171411) [ 2308.963195] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_550 Need at least 2 local interfaces for multi-rail [ 2310.858568] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 02:43:34 (1786171414) [ 2318.873080] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 02:43:42 (1786171422) [ 2325.056425] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 02:43:49 (1786171429) [ 2331.770624] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 02:43:55 (1786171435) [ 2338.196540] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 02:44:02 (1786171442) [ 2345.435677] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 02:44:09 (1786171449) [ 2352.405936] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 02:44:16 (1786171456) [ 2359.218255] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 02:44:23 (1786171463) [ 2365.439415] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 02:44:29 (1786171469) [ 2372.728458] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 02:44:36 (1786171476) [ 2380.776829] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 02:44:44 (1786171484) [ 2388.739388] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 02:44:52 (1786171492) [ 2396.419897] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 02:45:00 (1786171500) [ 2402.814142] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 02:45:06 (1786171506) [ 2409.630877] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 02:45:13 (1786171513) [ 2419.993629] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 02:45:23 (1786171523) [ 2428.704517] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 02:45:32 (1786171532) [ 2435.731627] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 02:45:39 (1786171539) [ 2435.943318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2435.991409] LNet: 136752:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2436.007526] LNetError: 136752:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2436.032377] LNet: Removed LNI 192.168.201.22@tcp2 [ 2436.063964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2436.184529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2436.201847] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 2437.229178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2437.255459] LNet: 136894:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2437.260983] LNetError: 136894:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2437.284698] LNet: Removed LNI 192.168.201.22@tcp [ 2437.302173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2437.380354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2437.409112] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 2443.753692] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 02:45:47 (1786171547) [ 2450.971726] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 02:45:54 (1786171554) [ 2458.570270] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 02:46:02 (1786171562) [ 2466.361709] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 02:46:09 (1786171569) [ 2472.958515] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 02:46:16 (1786171576) [ 2480.772365] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 02:46:24 (1786171584) [ 2487.592223] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 02:46:31 (1786171591) [ 2495.070268] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 02:46:38 (1786171598) [ 2504.362141] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 02:46:47 (1786171607) [ 2512.294159] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 02:46:56 (1786171616) [ 2520.748421] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 02:47:04 (1786171624) [ 2530.165693] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 02:47:13 (1786171633) [ 2537.451404] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 2317 sec ========== 02:47:21 (1786171641) [ 2538.036544] LNet: 144364:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2538.045963] LNetError: 144364:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2538.062800] LNet: Removed LNI 192.168.201.22@tcp [ 2539.052862] Key type .llcrypt unregistered [ 2539.056095] Key type ._llcrypt unregistered [ 2552.589487] Key type ._llcrypt registered [ 2552.591074] Key type .llcrypt registered [ 2553.527119] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2553.554164] alg: No test for adler32 (adler32-zlib) [ 2554.589189] Lustre: Lustre: Build Version: 2.17.56_50_gc905d6b [ 2554.863378] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 2556.544247] Key type lgssc registered [ 2558.054940] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2631.104297] Lustre: Mounted lustre-client - version 2.17.56_50_gc905d6b [ 2636.622323] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2646.211234] Lustre: Unmounted lustre-client [ 2672.786208] Key type lgssc unregistered [ 2673.136787] LNet: 147325:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2673.155792] LNetError: 147325:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2673.211244] LNet: Removed LNI 192.168.201.22@tcp [ 2674.207865] Key type .llcrypt unregistered [ 2674.210904] Key type ._llcrypt unregistered