[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 450702748 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.002392] x2apic enabled [ 0.003011] Switched APIC routing to physical x2apic. [ 0.004020] kvm-guest: setup PV IPIs [ 0.007319] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008025] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010017] pid_max: default: 32768 minimum: 301 [ 0.011163] LSM: Security Framework initializing [ 0.012086] Yama: becoming mindful. [ 0.013056] SELinux: Initializing. [ 0.014077] *** VALIDATE selinux *** [ 0.023868] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.028456] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.029217] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030142] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031160] *** VALIDATE tmpfs *** [ 0.033576] *** VALIDATE proc *** [ 0.035227] *** VALIDATE cgroup *** [ 0.036016] *** VALIDATE cgroup2 *** [ 0.038162] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.039186] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.040015] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.041046] Spectre V2 : User space: Vulnerable [ 0.042017] Speculative Store Bypass: Vulnerable [ 0.045353] debug: unmapping init [mem 0xffffffff8f259000-0xffffffff8f260fff] [ 0.047242] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.048833] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.049032] ... version: 2 [ 0.050021] ... bit width: 48 [ 0.051024] ... generic registers: 4 [ 0.052021] ... value mask: 0000ffffffffffff [ 0.053022] ... max period: 00007fffffffffff [ 0.054022] ... fixed-purpose events: 3 [ 0.055020] ... event mask: 000000070000000f [ 0.057320] rcu: Hierarchical SRCU implementation. [ 0.059815] smp: Bringing up secondary CPUs ... [ 0.060709] x86: Booting SMP configuration: [ 0.061053] .... node #0, CPUs: #1 #2 #3 [ 0.065051] smp: Brought up 1 node, 4 CPUs [ 0.067024] smpboot: Max logical packages: 1 [ 0.068023] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.226943] node 0 deferred pages initialised in 155ms [ 0.229018] devtmpfs: initialized [ 0.230302] x86/mm: Memory block size: 128MB [ 0.232853] gcov: version magic: 0x41383552 [ 0.234403] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.235104] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.236365] pinctrl core: initialized pinctrl subsystem [ 0.237261] [ 0.237843] ************************************************************* [ 0.238025] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.239020] ** ** [ 0.240021] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.241022] ** ** [ 0.242017] ** This means that this kernel is built to expose internal ** [ 0.243018] ** IOMMU data structures, which may compromise security on ** [ 0.244016] ** your system. ** [ 0.245020] ** ** [ 0.246022] ** If you see this message and you are not debugging the ** [ 0.247019] ** kernel, report this immediately to your vendor! ** [ 0.248019] ** ** [ 0.249021] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.250022] ************************************************************* [ 0.251965] NET: Registered protocol family 16 [ 0.252551] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.253082] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.254099] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.256113] cpuidle: using governor menu [ 0.257720] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.260591] PCI: Using configuration type 1 for base access [ 0.263128] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.272141] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.273066] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.274240] cryptd: max_cpu_qlen set to 1000 [ 0.277256] ACPI: Added _OSI(Module Device) [ 0.278072] ACPI: Added _OSI(Processor Device) [ 0.280018] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.282018] ACPI: Added _OSI(Processor Aggregator Device) [ 0.287317] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.294412] ACPI: Interpreter enabled [ 0.296075] ACPI: PM: (supports S0 S3 S4 S5) [ 0.297014] ACPI: Using IOAPIC for interrupt routing [ 0.299241] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.303484] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.314061] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.316052] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.319022] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.322074] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.328560] acpiphp: Slot [2] registered [ 0.330158] acpiphp: Slot [5] registered [ 0.332204] acpiphp: Slot [6] registered [ 0.334120] acpiphp: Slot [3] registered [ 0.335106] acpiphp: Slot [4] registered [ 0.337095] acpiphp: Slot [7] registered [ 0.338111] acpiphp: Slot [8] registered [ 0.340212] acpiphp: Slot [9] registered [ 0.341125] acpiphp: Slot [10] registered [ 0.343185] acpiphp: Slot [11] registered [ 0.344107] acpiphp: Slot [12] registered [ 0.346120] acpiphp: Slot [13] registered [ 0.347102] acpiphp: Slot [14] registered [ 0.349152] acpiphp: Slot [15] registered [ 0.350106] acpiphp: Slot [16] registered [ 0.352116] acpiphp: Slot [17] registered [ 0.353122] acpiphp: Slot [18] registered [ 0.355125] acpiphp: Slot [19] registered [ 0.356111] acpiphp: Slot [20] registered [ 0.358075] acpiphp: Slot [21] registered [ 0.359112] acpiphp: Slot [22] registered [ 0.360117] acpiphp: Slot [23] registered [ 0.361195] acpiphp: Slot [24] registered [ 0.363117] acpiphp: Slot [25] registered [ 0.364102] acpiphp: Slot [26] registered [ 0.366125] acpiphp: Slot [27] registered [ 0.367128] acpiphp: Slot [28] registered [ 0.369128] acpiphp: Slot [29] registered [ 0.370111] acpiphp: Slot [30] registered [ 0.371139] acpiphp: Slot [31] registered [ 0.373061] PCI host bridge to bus 0000:00 [ 0.374018] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.376023] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.377018] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.379023] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.382024] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.384028] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.386200] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.387870] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.391213] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.399788] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.406045] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.408021] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.411022] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.413020] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.416702] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.419850] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.422059] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.425991] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.431018] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.441791] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.445670] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.452938] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.460018] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.466018] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.483026] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.494354] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.504021] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.512022] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.533021] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.548634] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.551400] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.553382] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.556450] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.558224] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.562197] iommu: Default domain type: Passthrough [ 0.564455] SCSI subsystem initialized [ 0.566160] ACPI: bus type USB registered [ 0.568146] usbcore: registered new interface driver usbfs [ 0.570080] usbcore: registered new interface driver hub [ 0.572092] usbcore: registered new device driver usb [ 0.574187] pps_core: LinuxPPS API ver. 1 registered [ 0.575012] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.578072] PTP clock support registered [ 0.581142] EDAC MC: Ver: 3.0.0 [ 0.582399] PCI: Using ACPI for IRQ routing [ 0.584098] NetLabel: Initializing [ 0.585019] NetLabel: domain hash size = 128 [ 0.587026] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.589103] NetLabel: unlabeled traffic allowed by default [ 0.591163] vgaarb: loaded [ 0.593175] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.595020] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.601423] clocksource: Switched to clocksource kvm-clock [ 0.712283] VFS: Disk quotas dquot_6.6.0 [ 0.714116] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.716820] *** VALIDATE ramfs *** [ 0.718213] *** VALIDATE hugetlbfs *** [ 0.720759] pnp: PnP ACPI init [ 0.723283] pnp: PnP ACPI: found 6 devices [ 0.747529] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.751303] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.753599] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.755911] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.758589] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.761194] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.764200] NET: Registered protocol family 2 [ 0.766808] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.771711] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.775530] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.781237] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.785324] TCP: Hash tables configured (established 65536 bind 65536) [ 0.788316] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.791149] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.793441] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.796599] NET: Registered protocol family 1 [ 0.799455] RPC: Registered named UNIX socket transport module. [ 0.801873] RPC: Registered udp transport module. [ 0.803790] RPC: Registered tcp transport module. [ 0.805679] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.808071] NET: Registered protocol family 44 [ 0.809845] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.812258] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.814685] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.817186] PCI: CLS 0 bytes, default 64 [ 0.819054] Unpacking initramfs... [ 2.272638] debug: unmapping init [mem 0xffff9732fcc64000-0xffff9732fffcffff] [ 2.276763] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.279298] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.282336] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.816926] Initialise system trusted keyrings [ 2.820534] Key type blacklist registered [ 2.822838] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.833741] zbud: loaded [ 2.837809] *** VALIDATE nfs *** [ 2.839533] *** VALIDATE nfs4 *** [ 2.841448] pstore: using deflate compression [ 2.845158] Platform Keyring initialized [ 2.964441] NET: Registered protocol family 38 [ 2.966625] Key type asymmetric registered [ 2.968594] Asymmetric key parser 'x509' registered [ 2.970871] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.974764] io scheduler mq-deadline registered [ 2.976491] io scheduler kyber registered [ 2.978422] io scheduler bfq registered [ 2.980765] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.984494] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.987390] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.990432] ACPI: Power Button [PWRF] [ 2.996510] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.003441] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.035159] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.065048] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.094556] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.100286] Non-volatile memory driver v1.3 [ 3.102134] Linux agpgart interface v0.103 [ 3.136749] virtio_blk virtio1: [vda] 149968 512-byte logical blocks (76.8 MB/73.2 MiB) [ 3.139737] vda: detected capacity change from 0 to 76783616 [ 3.156602] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.160026] vdb: detected capacity change from 0 to 1073741824 [ 3.169041] libphy: Fixed MDIO Bus: probed [ 3.176451] usbcore: registered new interface driver usbserial_generic [ 3.179271] usbserial: USB Serial support registered for generic [ 3.182262] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.186532] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.188103] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.191190] mousedev: PS/2 mouse device common for all mice [ 3.194565] rtc_cmos 00:05: RTC can wake from S4 [ 3.196212] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.200292] rtc_cmos 00:05: registered as rtc0 [ 3.209028] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.209425] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.214055] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.217576] intel_pstate: CPU model not supported [ 3.224145] hid: raw HID events driver (C) Jiri Kosina [ 3.226536] usbcore: registered new interface driver usbhid [ 3.228263] usbhid: USB HID core driver [ 3.229688] drop_monitor: Initializing network drop monitor service [ 3.231448] Initializing XFRM netlink socket [ 3.232924] NET: Registered protocol family 10 [ 3.235603] Segment Routing with IPv6 [ 3.236973] NET: Registered protocol family 17 [ 3.239192] mpls_gso: MPLS GSO support [ 3.244371] RAS: Correctable Errors collector initialized. [ 3.247189] AVX version of gcm_enc/dec engaged. [ 3.248896] AES CTR mode by8 optimization enabled [ 3.328752] sched_clock: Marking stable (3328667748, 0)->(4302847568, -974179820) [ 3.333133] registered taskstats version 1 [ 3.335312] Loading compiled-in X.509 certificates [ 3.337857] zswap: loaded using pool lzo/zbud [ 3.365297] Key type big_key registered [ 3.378906] Key type encrypted registered [ 3.380950] ima: No TPM chip found, activating TPM-bypass! [ 3.383188] ima: Allocated hash algorithm: sha1 [ 3.385177] ima: No architecture policies found [ 3.386899] evm: Initialising EVM extended attributes: [ 3.389099] evm: security.selinux [ 3.390392] evm: security.ima [ 3.391341] evm: security.capability [ 3.392525] evm: HMAC attrs: 0x1 [ 3.395257] rtc_cmos 00:05: setting system clock to 2026-09-10 18:40:57 UTC (1789065657) [ 3.402072] debug: unmapping init [mem 0xffffffff90203000-0xffffffff903fffff] [ 3.405533] debug: unmapping init [mem 0xffffffff8ef82000-0xffffffff8f258fff] [ 3.418374] Write protecting the kernel read-only data: 28672k [ 3.421883] debug: unmapping init [mem 0xffffffff8d603000-0xffffffff8d7fffff] [ 3.425073] debug: unmapping init [mem 0xffffffff8df14000-0xffffffff8dffffff] [ 3.458977] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.467429] systemd[1]: Detected virtualization kvm. [ 3.469129] systemd[1]: Detected architecture x86-64. [ 3.471118] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.498549] systemd[1]: No hostname configured. [ 3.500484] systemd[1]: Set hostname to . [ 3.502212] random: systemd: uninitialized urandom read (16 bytes read) [ 3.505091] systemd[1]: Initializing machine ID from random generator. [ 3.640333] random: systemd: uninitialized urandom read (16 bytes read) [ 3.645414] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 3.650254] random: systemd: uninitialized urandom read (16 bytes read) [ 3.653021] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 3.658538] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... Starting Create Volatile Files and Directories... [ OK ] Started Memstrack Anylazing Service. Starting Setup Virtual Console... [ OK ] Reached target Swap. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... Starting Apply Kernel Variables... [ OK ] Listening on udev Control Socket. [ OK ] Reached target Timers. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Slices. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.270773] device-mapper: uevent: version 1.0.3 [ 4.273966] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.025016] random: fast init done [ 5.037367] virtio_net virtio0 ens2: renamed from eth0 [ 5.094443] scsi host0: ata_piix [ 5.144517] scsi host1: ata_piix [ 5.146547] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 5.148642] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 9.334620] dracut-initqueue[583]: RTNETLINK answers: File exists [ 9.903090] random: crng init done [ 9.904787] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.293846] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Swap. [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. Stopping udev Kernel Device Manager... [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.422798] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.692595] SELinux: Disabled at runtime. [ 11.756111] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.763780] systemd[1]: Detected virtualization kvm. [ 11.765482] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.275896] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.278649] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.282168] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.286383] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.289841] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.299137] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.305590] systemd[1]: Starting Remount Root and Kernel File Systems... Starting Remount Root and Kernel File Systems... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Activating swap /dev/disk/by-label/SWAP... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Listening on udev Kernel Socket. [ OK ] Created slice system-serial\x2dgetty.slice. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on initctl Compatibility Named Pipe. Mounting POSIX Message Queue File System... [ OK ] Created slice system-getty.slice. [ 12.369306] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Created slice system-sshd\x2dkeygen.slice. Mounting Huge Pages File System... Starting Apply Kernel Variables... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Mounting Kernel Debug File System... [ OK ] Listening on Process Core Dump Socket. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Stopped target Initrd File Systems. [ OK ] Started Journal Service. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Kernel Debug File System. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 12.733791] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.046558] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.086206] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.151147] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.171940] EDAC sbridge: Ver: 1.1.2 [ 14.189112] Key type dns_resolver registered [ 14.499851] NFS: Registering the id_resolver key type [ 14.501904] Key type id_resolver registered [ 14.504386] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. [ OK ] Started D-Bus System Message Bus. [ OK ] Reached target sshd-keygen.target. Starting Network Manager... Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Login Service... [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Started OpenSSH server daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg452-client login: [ 41.368252] libcfs: loading out-of-tree module taints kernel. [ 41.416091] Key type ._llcrypt registered [ 41.420416] Key type .llcrypt registered [ 41.703466] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 41.711195] alg: No test for adler32 (adler32-zlib) [ 42.735231] Lustre: Lustre: Build Version: 2.17.58_40_g706ff57 [ 43.097093] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 44.728322] Key type lgssc registered [ 45.475888] Lustre: Echo OBD driver; http://www.lustre.org/ [ 198.431400] Lustre: Mounted lustre-client - version 2.17.58_40_g706ff57 [ 204.450566] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 224.200532] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing check_logdir /tmp/testlogs/ [ 224.232181] Lustre: lustre-OST0000-osc-ffff973344c3e000: disconnect after 23s idle [ 232.309901] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing yml_node [ 237.642370] Lustre: DEBUG MARKER: Client: 2.17.58.40 [ 240.892576] Lustre: DEBUG MARKER: MDS: 2.17.58.40 [ 244.056416] Lustre: DEBUG MARKER: OSS: 2.17.58.40 [ 246.051903] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Thu Sep 10 14:44:58 EDT 2026 [ 250.014113] hrtimer: interrupt took 5321089 ns [ 262.912163] Lustre: Unmounted lustre-client [ 301.122450] Key type lgssc unregistered [ 301.418171] LNet: 8433:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 301.433314] LNetError: 8433:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 301.500338] LNet: Removed LNI 192.168.204.52@tcp [ 302.381186] Key type .llcrypt unregistered [ 302.383288] Key type ._llcrypt unregistered [ 338.393429] Key type ._llcrypt registered [ 338.395365] Key type .llcrypt registered [ 338.606888] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 338.627829] alg: No test for adler32 (adler32-zlib) [ 339.530790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 339.581939] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 339.600666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 344.156601] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 344.422113] LNet: 9317:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 344.441067] LNetError: 9317:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 344.457674] LNet: Removed LNI 192.168.204.52@tcp [ 345.096944] Key type .llcrypt unregistered [ 345.101301] Key type ._llcrypt unregistered [ 353.878035] Key type ._llcrypt registered [ 353.882969] Key type .llcrypt registered [ 355.520461] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 14:46:47 (1789066007) [ 355.985474] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 356.021800] alg: No test for adler32 (adler32-zlib) [ 356.997243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 357.096487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 357.128231] LNet: 9726:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 357.146905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 357.197915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 357.226623] LNet: 9733:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 366.016458] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 14:46:58 (1789066018) [ 366.457703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 366.482519] LNet: 10312:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 366.511732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 366.634556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 373.942825] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 14:47:06 (1789066026) [ 374.283626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 374.301917] LNet: 10898:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 374.326901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 374.401820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 382.606450] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 14:47:14 (1789066034) [ 383.001657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 383.044888] LNet: 11482:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 383.084546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 383.217990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 391.518176] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 14:47:23 (1789066043) [ 391.726018] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 391.781928] LNet: 12062:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 391.834613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 391.969395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 392.640592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 400.541234] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 14:47:33 (1789066053) [ 400.984220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 401.003487] LNet: 12657:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 401.040292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 401.158558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 401.705434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 402.177299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 409.881525] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 14:47:42 (1789066062) [ 410.123077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 410.152551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 410.231371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 417.407176] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 14:47:49 (1789066069) [ 417.702878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 417.729766] LNet: 13838:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 417.739345] LNet: 13838:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 417.756386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 417.835048] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 417.877482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 418.284696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 418.317722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 418.850735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 418.910584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 419.372520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 419.442752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 420.084884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 420.167311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 420.618294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 420.668738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 421.323919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 421.391914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 422.019046] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 422.060230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 422.633520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 422.712071] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 423.339906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 423.378578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 423.885433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 423.941760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 424.495870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 424.562579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 425.178546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 425.235425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 432.602505] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 14:48:05 (1789066085) [ 432.848229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 432.916938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 433.055524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 433.132441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 441.274973] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 14:48:13 (1789066093) [ 441.586320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 441.681190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 441.786428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 441.832948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 449.195483] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 14:48:21 (1789066101) [ 449.489902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 449.538694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 449.674624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 449.728844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 457.094674] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 14:48:29 (1789066109) [ 457.355532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 457.382574] LNet: 16280:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 457.394797] LNet: 16280:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 457.444024] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 457.525543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 457.612747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 464.741165] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 14:48:37 (1789066117) [ 464.886798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 464.958276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 465.098154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 465.163887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 473.587281] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 14:48:45 (1789066125) [ 473.862344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 473.928672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 474.099524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 474.148082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 482.494657] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 14:48:55 (1789066135) [ 482.863665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 482.901446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 482.987932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 483.609218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 491.422466] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 14:49:03 (1789066143) [ 491.819467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 491.864972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 492.012338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 492.250265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 499.962728] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 14:49:12 (1789066152) [ 500.339454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 500.374801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 500.549895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 500.915786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 507.829404] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 14:49:20 (1789066160) [ 508.072066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 508.115875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 508.614667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 508.972158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 517.101130] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 14:49:29 (1789066169) [ 517.356288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 517.406435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 517.473357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 517.528112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 526.040910] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 14:49:38 (1789066178) [ 526.405774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 526.436640] LNet: 21116:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 526.445574] LNet: 21116:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 7 previous similar messages [ 526.481716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 526.552446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 526.593249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 534.406387] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 14:49:47 (1789066187) [ 534.564861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 534.618880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 534.689638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 542.495871] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 14:49:54 (1789066194) [ 542.708797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 542.782513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 542.897847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 549.889721] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 14:50:02 (1789066202) [ 550.061631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 550.098964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 550.164578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 550.464425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 557.314758] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 14:50:10 (1789066210) [ 557.776537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 557.833946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 557.933847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 557.963955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 564.924970] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 14:50:17 (1789066217) [ 565.273835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 565.336840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 565.427710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 565.453748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 572.544208] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 14:50:25 (1789066225) [ 572.687794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 572.748272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 572.918973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 572.964274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 581.826384] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 14:50:34 (1789066234) [ 582.246848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 582.300904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 582.394571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 582.480819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 590.317569] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 14:50:43 (1789066243) [ 590.670857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 590.730526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 590.851801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 590.944415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 590.973633] LNetError: 25842:0:(peer.c:2366:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 591.042742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 591.085212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 598.891786] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 14:50:51 (1789066251) [ 599.041922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 599.164345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 599.309944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.jfzRHBLqiW [ 599.411977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.jfzRHBLqiW [ 599.509798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.jfzRHBLqiW [ 606.238176] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 14:50:58 (1789066258) [ 606.901605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 606.998145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 607.051184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 607.083415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 607.135992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 613.623203] Lustre: DEBUG MARKER: == sanity-lnet test 40: LNetAddPeer merges a pre-existing non-primary peer NID ========================================================== 14:51:06 (1789066266) [ 614.020195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 614.076956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 614.173303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 614.207606] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 615.763277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.52@tcp [ 617.167769] Lustre: Lustre: Build Version: 2.17.58_40_g706ff57 [ 619.192255] Key type lgssc registered [ 620.506461] Lustre: Echo OBD driver; http://www.lustre.org/ [ 624.752449] LNetError: 27604:0:(lib-move.c:2151:lnet_handle_find_routed_path()) peer 192.168.204.52@tcp99 has no available nets [ 627.552229] Key type lgssc unregistered [ 627.912255] LNetError: 28710:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 627.929651] LNet: Removed LNI 192.168.204.52@tcp [ 628.508368] Key type .llcrypt unregistered [ 628.514665] Key type ._llcrypt unregistered [ 635.349275] Key type ._llcrypt registered [ 635.354958] Key type .llcrypt registered [ 636.777068] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 14:51:29 (1789066289) [ 637.668354] Key type .llcrypt unregistered [ 637.670219] Key type ._llcrypt unregistered [ 645.427685] Key type ._llcrypt registered [ 645.432346] Key type .llcrypt registered [ 645.621959] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 645.646573] alg: No test for adler32 (adler32-zlib) [ 646.737936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 646.856747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 646.913507] LNet: 29620:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=16 less than recommended minimum 512 [ 646.925213] LNet: 29620:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 [ 646.939496] LNet: 29620:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 [ 646.959937] LNet: Message forwarding will be enabled in 2s [ 650.080139] LNet: Message forwarding enabled [ 650.616583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 650.642592] LNet: Message forwarding will stop in 2s to 6s [ 654.183531] LNet: Message forwarding disabled [ 655.239658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 655.276763] LNet: 29719:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=16 less than recommended minimum 512 [ 655.290609] LNet: 29719:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 [ 655.304161] LNet: 29719:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 [ 655.323247] LNet: Message forwarding will be enabled in 2s [ 655.519549] LNet: 29732:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 656.215560] Key type .llcrypt unregistered [ 656.217847] Key type ._llcrypt unregistered [ 669.200881] Key type ._llcrypt registered [ 669.204325] Key type .llcrypt registered [ 670.291624] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 14:52:03 (1789066323) [ 671.071843] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 671.113776] alg: No test for adler32 (adler32-zlib) [ 672.076788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 672.164787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 679.392917] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 14:52:11 (1789066331) [ 679.642316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 679.667279] LNet: 31134:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 679.705852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 679.769802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 679.849512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 679.887614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 679.922385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 679.963354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 680.001185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 680.051247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 680.089618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 680.142039] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 680.173756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 680.208761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 680.243543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 680.274723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 680.316077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 680.361071] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 680.414761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 680.503033] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 680.555983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 680.614340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 680.673859] LNetError: 31177:0:(router.c:625:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 686.992881] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 14:52:19 (1789066339) [ 687.114568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 687.135520] LNet: 31746:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 687.160808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 687.260912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 695.951492] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 14:52:28 (1789066348) [ 698.067732] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg452-server [ 700.037847] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 14:52:32 (1789066352) [ 700.152385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 700.184106] LNet: 32674:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 700.214823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 700.285048] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 700.310807] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 701.308246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.204.[53-57]@tcp [ 701.339341] LNet: 32814:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 701.421347] LNetError: 32683:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.204.53@tcp is being used as a gateway but routing feature is not turned on [ 701.442753] LNetError: 32683:0:(lib-lnet.h:1281:lnet_set_route_aliveness()) route to tcp1 through 192.168.204.53@tcp has gone from up to down [ 708.752356] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 14:52:41 (1789066361) [ 710.997130] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg452-server [ 712.860668] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 14:52:45 (1789066365) [ 713.198584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 713.241199] LNet: 33751:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 713.254753] LNetError: 33751:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 713.295845] LNet: Removed LNI 192.168.204.52@tcp [ 713.323286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 713.407038] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 713.450951] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 714.120076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.204.[53-58/2]@tcp [ 714.144225] LNet: 33868:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 714.153313] LNet: 33868:0:(router.c:729:lnet_add_route()) Skipped 4 previous similar messages [ 714.171478] LNetError: 33755:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.204.53@tcp is being used as a gateway but routing feature is not turned on [ 714.184466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.204.[53-58/2]@tcp [ 714.188083] LNetError: 33755:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 714.188107] LNetError: 33755:0:(lib-lnet.h:1281:lnet_set_route_aliveness()) route to tcp103 through 192.168.204.53@tcp has gone from up to down [ 714.188112] LNetError: 33755:0:(lib-lnet.h:1281:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 721.959752] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 14:52:54 (1789066374) [ 722.128215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 722.184833] LNet: 34456:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 722.197664] LNetError: 34456:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 722.260526] LNet: Removed LNI 192.168.204.52@tcp [ 722.324780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 722.517411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 722.655458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 722.713954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 722.833372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 722.933804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 723.002001] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 723.135595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 723.248734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 723.298409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 723.385715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 723.492436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 723.604646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 723.739725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 723.869552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 723.949648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 724.050957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 724.202238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 724.272564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 724.379447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 724.512017] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 724.600879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 724.750694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 724.969484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 725.070443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 725.156705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 725.310767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 725.398773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 725.560260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 725.699760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 725.736358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 725.806874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 725.874258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 725.932824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 725.994517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 734.253526] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 14:53:06 (1789066386) [ 736.464919] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg452-server [ 738.392340] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 14:53:11 (1789066391) [ 740.662340] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg452-server [ 742.494525] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 14:53:15 (1789066395) [ 742.904784] LNet: 35862:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 742.911116] LNet: 35862:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 11 previous similar messages [ 743.653685] Key type .llcrypt unregistered [ 743.659149] Key type ._llcrypt unregistered [ 751.869094] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 752.040102] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 752.042420] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 752.062591] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 754.704251] Key type ._llcrypt registered [ 754.705679] Key type .llcrypt registered [ 754.862804] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 754.873542] alg: No test for adler32 (adler32-zlib) [ 755.817922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 755.893718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 755.917239] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 756.791195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 756.833826] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 759.427849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 759.479395] LNet: Removed LNI 10.1.2.3@tcp [ 759.749411] LNet: 36579:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 759.759786] LNetError: 36579:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 759.780419] LNet: Removed LNI 192.168.204.52@tcp [ 760.596512] Key type .llcrypt unregistered [ 760.599340] Key type ._llcrypt unregistered [ 777.819218] Key type ._llcrypt registered [ 777.845895] Key type .llcrypt registered [ 779.878901] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 14:53:51 (1789066431) [ 780.950495] Key type .llcrypt unregistered [ 780.963677] Key type ._llcrypt unregistered [ 791.901986] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 792.243028] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 792.258548] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 792.273848] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 795.138222] Key type ._llcrypt registered [ 795.140666] Key type .llcrypt registered [ 795.514693] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 795.559920] alg: No test for adler32 (adler32-zlib) [ 796.531697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 796.631653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 796.678780] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 797.902478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 797.937607] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 801.792788] LNet: 38074:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 801.809509] LNetError: 38074:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 801.823946] LNet: Removed LNI 10.1.2.3@tcp [ 801.853422] LNet: Removed LNI 192.168.204.52@tcp [ 802.582737] Key type .llcrypt unregistered [ 802.586497] Key type ._llcrypt unregistered [ 819.342318] Key type ._llcrypt registered [ 819.348199] Key type .llcrypt registered [ 821.018834] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 14:54:33 (1789066473) [ 821.566410] Key type .llcrypt unregistered [ 821.570945] Key type ._llcrypt unregistered [ 829.452063] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 829.625964] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 829.632254] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 829.647431] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 832.353035] Key type ._llcrypt registered [ 832.354969] Key type .llcrypt registered [ 832.767627] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 832.802295] alg: No test for adler32 (adler32-zlib) [ 833.743695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 833.799615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 833.835899] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 836.079196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 836.123264] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 838.726058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 838.773472] LNet: Removed LNI 10.1.2.3@tcp [ 838.796934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 838.824981] LNet: Removed LNI 10.1.2.31@tcp [ 838.844808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 838.871975] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 839.615946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 839.654690] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 840.778943] LNet: 39990:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 840.786235] LNetError: 39990:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg:0 not found [ 840.795707] LNet: Removed LNI 10.1.2.31@tcp [ 840.824231] LNet: Removed LNI 10.1.2.3@tcp [ 841.699590] Key type .llcrypt unregistered [ 841.702119] Key type ._llcrypt unregistered [ 855.077846] Key type ._llcrypt registered [ 855.087925] Key type .llcrypt registered [ 856.823282] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 14:55:08 (1789066508) [ 857.234840] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 857.275733] alg: No test for adler32 (adler32-zlib) [ 858.315781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 858.511830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 858.577210] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 859.490191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 859.503339] LNet: 40941:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 859.508712] LNetError: 40941:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 859.524193] LNet: Removed LNI 192.168.204.52@tcp [ 859.556301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 859.606385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.204.52@tcp [ 859.632205] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 860.288962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.204.52@tcp [ 860.334539] LNet: Removed LNI 192.168.204.52@tcp [ 860.395584] LNet: 41076:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 861.084507] Key type .llcrypt unregistered [ 861.086177] Key type ._llcrypt unregistered [ 873.705172] Key type ._llcrypt registered [ 873.707837] Key type .llcrypt registered [ 875.056413] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 14:55:27 (1789066527) [ 877.245489] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg452-server [ 879.045434] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 14:55:31 (1789066531) [ 879.558119] Key type .llcrypt unregistered [ 879.559968] Key type ._llcrypt unregistered [ 885.953969] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 886.107610] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 886.111667] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 886.116553] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 888.729286] Key type ._llcrypt registered [ 888.731056] Key type .llcrypt registered [ 888.876801] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 888.889087] alg: No test for adler32 (adler32-zlib) [ 889.764520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 889.815673] LNet: Added LNI 192.168.204.52@tcp1 [8/256/0/180] [ 890.345948] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 892.182689] LNet: 42907:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 892.189764] LNetError: 42907:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 892.196916] LNet: Removed LNI 192.168.204.52@tcp1 [ 892.206913] LNet: Removed LNI 10.1.2.3@tcp2 [ 892.761577] Key type .llcrypt unregistered [ 892.763548] Key type ._llcrypt unregistered [ 904.525771] Key type ._llcrypt registered [ 904.532706] Key type .llcrypt registered [ 906.154251] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 14:55:58 (1789066558) [ 906.599539] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 906.615674] alg: No test for adler32 (adler32-zlib) [ 907.590484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 907.634372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 907.662862] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 908.523950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 908.553399] LNet: 43876:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 908.558903] LNetError: 43876:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 908.576834] LNet: Removed LNI 192.168.204.52@tcp [ 908.593536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 908.653662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 908.686324] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 915.856416] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 14:56:08 (1789066568) [ 916.106887] LNet: 44467:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 916.117658] LNetError: 44467:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 916.137525] LNet: Removed LNI 192.168.204.52@tcp [ 916.759597] Key type .llcrypt unregistered [ 916.765673] Key type ._llcrypt unregistered [ 922.041291] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 922.176755] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 922.185233] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 922.195769] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 924.861467] Key type ._llcrypt registered [ 924.863608] Key type .llcrypt registered [ 925.061402] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 925.083640] alg: No test for adler32 (adler32-zlib) [ 925.999224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 926.063983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 926.084852] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 926.922923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 926.945231] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 928.959622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 928.978939] LNet: 45207:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 928.985959] LNetError: 45207:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 928.992248] LNet: Removed LNI 10.1.2.3@tcp [ 929.000977] LNet: Removed LNI 192.168.204.52@tcp [ 929.037604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 929.156980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 929.195350] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 929.202650] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 929.651902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 929.674788] LNet: 45234:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 929.682410] LNetError: 45234:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 929.688930] LNetError: 45234:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 929.696960] LNet: Removed LNI 10.1.2.3@tcp [ 929.708232] LNet: Removed LNI 192.168.204.52@tcp [ 929.730431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 929.790841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 929.816795] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 929.821446] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 930.330024] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 930.349411] LNet: Added LNI 192.168.204.52@tcp2 [8/256/0/180] [ 931.259656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 931.301502] LNet: 45396:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 931.318693] LNetError: 45396:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 931.324877] LNetError: 45396:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 931.337597] LNet: Removed LNI 10.1.2.3@tcp [ 931.340716] LNet: Removed LNI 192.168.204.52@tcp [ 931.353900] LNet: Removed LNI 192.168.204.52@tcp2 [ 931.380058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 931.455578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 931.485314] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 931.489660] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 931.495035] LNet: Added LNI 192.168.204.52@tcp2 [8/256/0/180] [ 939.579034] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 14:56:32 (1789066592) [ 939.710797] LNet: 45987:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 939.720281] LNetError: 45987:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 939.736075] LNetError: 45987:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 939.748335] LNet: Removed LNI 10.1.2.3@tcp [ 939.754960] LNet: Removed LNI 192.168.204.52@tcp [ 939.768545] LNet: Removed LNI 192.168.204.52@tcp2 [ 940.656770] Key type .llcrypt unregistered [ 940.661270] Key type ._llcrypt unregistered [ 947.638583] Key type ._llcrypt registered [ 947.646447] Key type .llcrypt registered [ 947.975974] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 948.003518] alg: No test for adler32 (adler32-zlib) [ 949.021397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 949.126813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 949.159801] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 950.195718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 950.214054] LNet: 46385:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 950.219783] LNetError: 46385:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 950.231563] LNet: Removed LNI 192.168.204.52@tcp [ 950.251737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 950.365710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 950.406824] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 951.233522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 951.258722] LNet: 46415:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 951.266540] LNetError: 46415:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 951.287408] LNet: Removed LNI 192.168.204.52@tcp [ 951.308789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 951.382093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 951.407040] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 951.971802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 952.014518] LNet: Removed LNI 192.168.204.52@tcp [ 952.034042] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 952.141674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 952.178357] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 952.908950] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 953.158622] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 953.168020] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 953.173520] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 955.691819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 955.730156] LNet: 46524:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 955.734058] LNet: 46524:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 955.742882] LNetError: 46524:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 955.750966] LNetError: 46524:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 955.772229] LNet: Removed LNI 192.168.204.52@tcp [ 955.801600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 955.891747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 955.923537] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 963.904962] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 14:56:56 (1789066616) [ 964.048779] LNet: 47113:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 964.065981] LNetError: 47113:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 964.080188] LNet: Removed LNI 192.168.204.52@tcp [ 964.955778] Key type .llcrypt unregistered [ 964.957959] Key type ._llcrypt unregistered [ 971.915411] Key type ._llcrypt registered [ 971.924737] Key type .llcrypt registered [ 972.240641] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 972.260147] alg: No test for adler32 (adler32-zlib) [ 973.348969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 973.416387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 973.461256] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 974.608343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 974.635238] LNet: 47517:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 974.641637] LNetError: 47517:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 974.668635] LNet: Removed LNI 192.168.204.52@tcp [ 974.695987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 974.791275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 974.843103] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 982.927768] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 14:57:15 (1789066635) [ 983.020139] LNet: 48102:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 983.031887] LNetError: 48102:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 983.088475] LNet: Removed LNI 192.168.204.52@tcp [ 983.946217] Key type .llcrypt unregistered [ 983.947960] Key type ._llcrypt unregistered [ 991.049578] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 991.322924] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 991.326388] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 991.335777] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 993.895740] Key type ._llcrypt registered [ 993.897316] Key type .llcrypt registered [ 994.153057] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 994.174271] alg: No test for adler32 (adler32-zlib) [ 995.084890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 995.171750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 995.204674] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 996.184750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 996.256631] LNet: 48556:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 996.266789] LNetError: 48556:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 996.291050] LNet: Removed LNI 192.168.204.52@tcp [ 996.352137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 996.439682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 996.472620] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 1003.783353] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 14:57:36 (1789066656) [ 1003.998587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1004.039077] LNet: 49156:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1004.047650] LNetError: 49156:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1004.067721] LNet: Removed LNI 192.168.204.52@tcp [ 1004.129801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1004.232143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 1010.864201] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 14:57:43 (1789066663) [ 1010.979921] LNet: 49729:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1011.697655] Key type .llcrypt unregistered [ 1011.699739] Key type ._llcrypt unregistered [ 1019.048188] Key type ._llcrypt registered [ 1019.051795] Key type .llcrypt registered [ 1019.273189] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1019.303827] alg: No test for adler32 (adler32-zlib) [ 1020.292550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1020.382897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1020.433289] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 1021.277990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1021.302143] LNet: 50132:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1021.310140] LNetError: 50132:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1021.329891] LNet: Removed LNI 192.168.204.52@tcp [ 1021.344457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1021.416706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 1021.439902] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 1028.742771] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 14:58:01 (1789066681) [ 1029.031480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1029.073430] LNet: 50725:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1029.083536] LNetError: 50725:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1029.119279] LNet: Removed LNI 192.168.204.52@tcp [ 1029.176269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1029.271661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 1035.469715] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 14:58:08 (1789066688) [ 1036.949348] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 1038.671673] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 14:58:11 (1789066691) [ 1040.197848] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 1042.245826] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 14:58:14 (1789066694) [ 1043.762616] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 1045.489571] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 14:58:18 (1789066698) [ 1047.161127] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 1048.745944] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 14:58:21 (1789066701) [ 1050.038119] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 1051.679696] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing ========================================================== 14:58:24 (1789066704) [ 1051.803287] LNet: 52743:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1052.375897] Key type .llcrypt unregistered [ 1052.380392] Key type ._llcrypt unregistered [ 1058.806117] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1059.077628] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1059.086459] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1059.100058] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1061.612762] Key type ._llcrypt registered [ 1061.617400] Key type .llcrypt registered [ 1061.867846] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1061.895903] alg: No test for adler32 (adler32-zlib) [ 1062.964048] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1063.023250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1063.038807] LNet: Added LNI 192.168.204.52@tcp [8/512/0/180] [ 1063.044088] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 1063.046985] LNet: 53083:0:(api-ni.c:2991:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1063.162644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1063.193596] LNet: 53102:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1063.202815] LNetError: 53102:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1063.213149] LNet: Removed LNI 10.1.2.3@tcp [ 1063.214845] LNetError: 53102:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1063.227697] LNet: Removed LNI 192.168.204.52@tcp [ 1063.267702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1063.352592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1063.396544] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 1063.468638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1063.486625] LNet: 53126:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1063.492385] LNetError: 53126:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1063.514433] LNet: Removed LNI 192.168.204.52@tcp [ 1063.533234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1063.607205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1063.621851] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 1063.631290] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1063.635981] LNet: 53134:0:(api-ni.c:2991:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1063.747739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1063.774700] LNet: 53152:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1063.786408] LNetError: 53152:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1063.802175] LNet: Removed LNI 10.1.2.3@tcp [ 1063.806038] LNetError: 53152:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1063.837933] LNet: Removed LNI 192.168.204.52@tcp [ 1063.876582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1063.971867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1064.019387] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 1064.026569] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1064.034594] LNet: 53160:0:(api-ni.c:2991:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1064.296546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1064.316781] LNet: 53186:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1064.327851] LNetError: 53186:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1064.340349] LNet: Removed LNI 10.1.2.3@tcp [ 1064.344506] LNetError: 53186:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1064.370992] LNet: Removed LNI 192.168.204.52@tcp [ 1064.408821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1064.487943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1064.525810] LNet: Added LNI 192.168.204.52@tcp [8/256/0/182] [ 1064.532304] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 1064.703348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1064.743470] LNet: 53214:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1064.751605] LNetError: 53214:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1064.759418] LNet: Removed LNI 10.1.2.3@tcp [ 1064.763692] LNetError: 53214:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1064.789346] LNet: Removed LNI 192.168.204.52@tcp [ 1064.805971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1064.912914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1064.973390] LNet: Added LNI 192.168.204.52@tcp [8/256/0/182] [ 1064.984971] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 1064.990774] LNet: 53222:0:(api-ni.c:2991:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1065.099512] LNet: Removed LNI 10.1.2.3@tcp [ 1065.120930] LNet: Removed LNI 192.168.204.52@tcp [ 1065.846069] Key type .llcrypt unregistered [ 1065.848954] Key type ._llcrypt unregistered [ 1078.509895] Key type ._llcrypt registered [ 1078.512882] Key type .llcrypt registered [ 1079.825588] Lustre: DEBUG MARKER: == sanity-lnet test 180: FastReg map error should not leak descriptors ========================================================== 14:58:52 (1789066732) [ 1081.716466] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_180 Need o2ib NETTYPE [ 1083.895238] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 14:58:56 (1789066736) [ 1084.460142] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1084.499629] alg: No test for adler32 (adler32-zlib) [ 1085.484798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1085.535186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1085.566218] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 1086.635035] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.204.52 [ 1086.661668] LNet: 54485:0:(config.c:1740:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1086.681297] LNet: Removed LNI 192.168.204.52@tcp [ 1093.870768] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 14:59:06 (1789066746) [ 1094.115821] LNet: 55055:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1094.122200] LNetError: 55055:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1094.135846] LNet: Removed LNI 192.168.204.52@tcp [ 1094.749800] Key type .llcrypt unregistered [ 1094.752397] Key type ._llcrypt unregistered [ 1104.092817] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1104.450336] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1106.189404] Key type ._llcrypt registered [ 1106.192877] Key type .llcrypt registered [ 1106.456857] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1106.478207] alg: No test for adler32 (adler32-zlib) [ 1107.880675] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1116.024076] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 14:59:28 (1789066768) [ 1116.111123] LNet: 55949:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1116.121634] LNetError: 55949:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1116.153684] LNet: Removed LNI 10.1.2.3@tcp [ 1116.966125] Key type .llcrypt unregistered [ 1116.969691] Key type ._llcrypt unregistered [ 1123.801708] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1124.143074] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1124.149357] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1124.160268] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1126.857848] Key type ._llcrypt registered [ 1126.860710] Key type .llcrypt registered [ 1127.066269] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1127.085700] alg: No test for adler32 (adler32-zlib) [ 1128.065366] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1135.107514] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 14:59:47 (1789066787) [ 1136.480586] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 1138.539164] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 14:59:51 (1789066791) [ 1138.765461] LNet: 57139:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1138.788044] LNetError: 57139:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1138.817190] LNet: Removed LNI 10.1.2.3@tcp [ 1139.420723] Key type .llcrypt unregistered [ 1139.424782] Key type ._llcrypt unregistered [ 1148.088297] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1148.451797] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1148.459426] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1148.464853] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1151.211849] Key type ._llcrypt registered [ 1151.215376] Key type .llcrypt registered [ 1151.471931] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1151.493828] alg: No test for adler32 (adler32-zlib) [ 1152.446948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1152.543522] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1163.190710] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 15:00:15 (1789066815) [ 1163.529545] LNet: 58369:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1163.544749] LNetError: 58369:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1163.564048] LNet: Removed LNI 10.1.2.3@tcp [ 1164.257937] Key type .llcrypt unregistered [ 1164.263787] Key type ._llcrypt unregistered [ 1171.136344] Key type ._llcrypt registered [ 1171.138793] Key type .llcrypt registered [ 1171.302659] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1171.323710] alg: No test for adler32 (adler32-zlib) [ 1172.242931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1172.265911] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 1179.967186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.152@tcp [ 1180.904155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp -d 192.168.204.152@tcp -m GET -e local_interrupt -r 1 [ 1180.936554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp -d 192.168.204.52@tcp -m GET -e local_interrupt -r 1 [ 1180.969824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.152@tcp [ 1180.990494] LNet: There was an unexpected network error while writing to 192.168.204.152: rc = -22 [ 1181.273626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1181.314659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1181.342713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1181.380598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1181.416705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1181.445375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1181.472476] LNet: 1 local NIs in recovery (showing 1): 192.168.204.52@tcp [ 1181.486854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1181.532859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1181.569155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1181.586289] LNet: 58807:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1181.613524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1181.663971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1181.718119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1181.769629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1181.810160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1181.858570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1181.933551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1182.647422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp -d 192.168.204.152@tcp -m GET -e local_dropped -r 1 [ 1182.699218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp -d 192.168.204.52@tcp -m GET -e local_dropped -r 1 [ 1182.752415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.152@tcp [ 1182.782354] LNet: There was an unexpected network error while writing to 192.168.204.152: rc = -22 [ 1182.797025] LNet: Skipped 1 previous similar message [ 1183.127988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1183.178686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1183.229647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1183.254941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1183.284453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1183.309687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1183.345832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1183.391284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1183.441629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1183.463794] LNet: 58918:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1183.487977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1183.517162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1183.547137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1183.577632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1183.607605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1183.637619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1183.680474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1184.200425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp -d 192.168.204.152@tcp -m GET -e local_aborted -r 1 [ 1184.234768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp -d 192.168.204.52@tcp -m GET -e local_aborted -r 1 [ 1184.267744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.152@tcp [ 1184.291142] LNet: There was an unexpected network error while writing to 192.168.204.152: rc = -22 [ 1184.520742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1184.569951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1184.614612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1184.652356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1184.683742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1184.712735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1184.747847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1184.782637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1184.814843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1184.838993] LNet: 59030:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1184.864920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1184.934232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1184.963068] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1185.000958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1185.034892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1185.061449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1185.090138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1185.713639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp -d 192.168.204.152@tcp -m GET -e local_no_route -r 1 [ 1185.747487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp -d 192.168.204.52@tcp -m GET -e local_no_route -r 1 [ 1185.778646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.152@tcp [ 1186.055791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1186.120688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1186.188827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1186.227746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1186.282048] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1186.353458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1186.420661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1186.485744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1186.553273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1186.607568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1186.668185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1186.711332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1186.743134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1186.791184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1186.847894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1186.892934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1187.553294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp -d 192.168.204.152@tcp -m GET -e local_timeout -r 1 [ 1187.632822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp -d 192.168.204.52@tcp -m GET -e local_timeout -r 1 [ 1187.698247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.152@tcp [ 1187.760355] LNet: There was an unexpected network error while writing to 192.168.204.152: rc = -22 [ 1187.768457] LNet: Skipped 2 previous similar messages [ 1188.021868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1188.093633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1188.142880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1188.189451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1188.232470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1188.291264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1188.322419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1188.354603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1188.397318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1188.424469] LNet: 59252:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1188.429051] LNet: 59252:0:(api-ni.c:301:recovery_interval_set()) Skipped 1 previous similar message [ 1188.452409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1188.500238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1188.550536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1188.596273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1188.632637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1188.674898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1188.705206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1189.440088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp -d 192.168.204.152@tcp -m GET -e local_error -r 1 [ 1189.497827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp -d 192.168.204.52@tcp -m GET -e local_error -r 1 [ 1189.549718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.152@tcp [ 1189.856745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1189.901587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1189.946042] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1189.968700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1190.023190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1190.078750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1190.115649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1190.153786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1190.182408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1190.232669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1190.266264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1190.305769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1190.361833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1190.409455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1190.438875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1190.483273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1190.740820] LNet: 59405:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1190.746183] LNetError: 59405:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1190.765212] LNet: Removed LNI 192.168.204.52@tcp [ 1191.318691] Key type .llcrypt unregistered [ 1191.323955] Key type ._llcrypt unregistered [ 1210.355558] Key type ._llcrypt registered [ 1210.358253] Key type .llcrypt registered [ 1211.836419] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 15:01:04 (1789066864) [ 1212.712420] Key type .llcrypt unregistered [ 1212.718963] Key type ._llcrypt unregistered [ 1218.475297] Key type ._llcrypt registered [ 1218.477280] Key type .llcrypt registered [ 1218.722520] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1218.764277] alg: No test for adler32 (adler32-zlib) [ 1219.724709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1219.768943] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 1227.411505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.152@tcp [ 1238.218817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1238.284651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1238.331743] LNet: Added LNI 192.168.204.52@tcp1 [8/256/0/180] [ 1240.301973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp -d 192.168.204.152@tcp -m GET -e local_interrupt -r 1 [ 1240.342863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp -d 192.168.204.152@tcp1 -m GET -e local_interrupt -r 1 [ 1240.376897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp -d 192.168.204.52@tcp -m GET -e local_interrupt -r 1 [ 1240.406105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp -d 192.168.204.52@tcp1 -m GET -e local_interrupt -r 1 [ 1240.441123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp1 -d 192.168.204.152@tcp -m GET -e local_interrupt -r 1 [ 1240.478845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp1 -d 192.168.204.152@tcp1 -m GET -e local_interrupt -r 1 [ 1240.520322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp1 -d 192.168.204.52@tcp -m GET -e local_interrupt -r 1 [ 1240.564186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp1 -d 192.168.204.52@tcp1 -m GET -e local_interrupt -r 1 [ 1240.593692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.152@tcp [ 1240.611204] LNet: There was an unexpected network error while writing to 192.168.204.152: rc = -22 [ 1240.618478] LNet: 1 local NIs in recovery (showing 1): 192.168.204.52@tcp [ 1240.922544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1240.992318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1241.043845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1241.109725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1241.139992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1241.180391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1241.229576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1241.261949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1241.320027] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1241.340418] LNet: 60898:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1241.359856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1241.390893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1241.447181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1241.503160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1241.532603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1241.585883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1241.641289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1242.315657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp -d 192.168.204.152@tcp -m GET -e local_dropped -r 1 [ 1242.371112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp -d 192.168.204.152@tcp1 -m GET -e local_dropped -r 1 [ 1242.460658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp -d 192.168.204.52@tcp -m GET -e local_dropped -r 1 [ 1242.545271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp -d 192.168.204.52@tcp1 -m GET -e local_dropped -r 1 [ 1242.606207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp1 -d 192.168.204.152@tcp -m GET -e local_dropped -r 1 [ 1242.660685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp1 -d 192.168.204.152@tcp1 -m GET -e local_dropped -r 1 [ 1242.712110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp1 -d 192.168.204.52@tcp -m GET -e local_dropped -r 1 [ 1242.755464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp1 -d 192.168.204.52@tcp1 -m GET -e local_dropped -r 1 [ 1242.827241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.152@tcp [ 1242.850631] LNet: There was an unexpected network error while writing to 192.168.204.152: rc = -22 [ 1242.866962] LNet: Skipped 2 previous similar messages [ 1243.137149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1243.185788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1243.246356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1243.322258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1243.362701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1243.417938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1243.454960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1243.492147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1243.528523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1243.551675] LNet: 61021:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1243.578711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1243.635686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1243.679107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1243.722680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1243.752600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1243.801677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1243.872149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1244.519047] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp -d 192.168.204.152@tcp -m GET -e local_aborted -r 1 [ 1244.566403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp -d 192.168.204.152@tcp1 -m GET -e local_aborted -r 1 [ 1244.604215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp -d 192.168.204.52@tcp -m GET -e local_aborted -r 1 [ 1244.658561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp -d 192.168.204.52@tcp1 -m GET -e local_aborted -r 1 [ 1244.702500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp1 -d 192.168.204.152@tcp -m GET -e local_aborted -r 1 [ 1244.746030] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp1 -d 192.168.204.152@tcp1 -m GET -e local_aborted -r 1 [ 1244.800348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp1 -d 192.168.204.52@tcp -m GET -e local_aborted -r 1 [ 1244.863914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp1 -d 192.168.204.52@tcp1 -m GET -e local_aborted -r 1 [ 1244.931914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.152@tcp [ 1244.965513] LNet: There was an unexpected network error while writing to 192.168.204.152: rc = -22 [ 1244.977143] LNet: Skipped 2 previous similar messages [ 1245.254970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1245.309679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1245.355617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1245.397625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1245.479350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1245.524371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1245.562328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1245.587318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1245.615760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1245.632147] LNet: 61145:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1245.651042] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1245.685644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1245.737584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1245.787840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1245.862617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1245.936129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1245.977088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1246.534759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp -d 192.168.204.152@tcp -m GET -e local_no_route -r 1 [ 1246.580796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp -d 192.168.204.152@tcp1 -m GET -e local_no_route -r 1 [ 1246.624264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp -d 192.168.204.52@tcp -m GET -e local_no_route -r 1 [ 1246.668662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp -d 192.168.204.52@tcp1 -m GET -e local_no_route -r 1 [ 1246.709169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp1 -d 192.168.204.152@tcp -m GET -e local_no_route -r 1 [ 1246.758405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp1 -d 192.168.204.152@tcp1 -m GET -e local_no_route -r 1 [ 1246.799987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp1 -d 192.168.204.52@tcp -m GET -e local_no_route -r 1 [ 1246.900371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp1 -d 192.168.204.52@tcp1 -m GET -e local_no_route -r 1 [ 1246.945479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.152@tcp [ 1246.966678] LNet: There was an unexpected network error while writing to 192.168.204.152: rc = -22 [ 1246.972870] LNet: Skipped 2 previous similar messages [ 1247.229823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1247.266991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1247.309761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1247.342782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1247.409249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1247.462314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1247.532822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1247.582434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1247.617860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1247.648825] LNet: 61268:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1247.692366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1247.728217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1247.767991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1247.806935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1247.835164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1247.880215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1247.919551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1248.528824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp -d 192.168.204.152@tcp -m GET -e local_timeout -r 1 [ 1248.581491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp -d 192.168.204.152@tcp1 -m GET -e local_timeout -r 1 [ 1248.631675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp -d 192.168.204.52@tcp -m GET -e local_timeout -r 1 [ 1248.681597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp -d 192.168.204.52@tcp1 -m GET -e local_timeout -r 1 [ 1248.747462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp1 -d 192.168.204.152@tcp -m GET -e local_timeout -r 1 [ 1248.807414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp1 -d 192.168.204.152@tcp1 -m GET -e local_timeout -r 1 [ 1248.870695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp1 -d 192.168.204.52@tcp -m GET -e local_timeout -r 1 [ 1248.910407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp1 -d 192.168.204.52@tcp1 -m GET -e local_timeout -r 1 [ 1248.979896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.152@tcp [ 1249.376960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1249.422959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1249.453487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1249.482687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1249.519821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1249.564265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1249.618857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1249.655833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1249.694715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1249.734676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1249.776712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1249.809631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1249.856316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1249.905444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1249.957968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1250.002679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1250.875394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp -d 192.168.204.152@tcp -m GET -e local_error -r 1 [ 1250.963951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp -d 192.168.204.152@tcp1 -m GET -e local_error -r 1 [ 1251.036251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp -d 192.168.204.52@tcp -m GET -e local_error -r 1 [ 1251.090572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp -d 192.168.204.52@tcp1 -m GET -e local_error -r 1 [ 1251.170430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp1 -d 192.168.204.152@tcp -m GET -e local_error -r 1 [ 1251.213846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp1 -d 192.168.204.152@tcp1 -m GET -e local_error -r 1 [ 1251.271627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp1 -d 192.168.204.52@tcp -m GET -e local_error -r 1 [ 1251.340912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp1 -d 192.168.204.52@tcp1 -m GET -e local_error -r 1 [ 1251.388939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.152@tcp [ 1251.418048] LNet: There was an unexpected network error while writing to 192.168.204.152: rc = -22 [ 1251.421994] LNet: Skipped 5 previous similar messages [ 1251.711827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1251.741288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1251.790193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1251.840629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1251.908640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1251.961410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1252.015520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1252.059948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1252.096987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1252.117166] LNet: 61515:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1252.123447] LNet: 61515:0:(api-ni.c:301:recovery_interval_set()) Skipped 1 previous similar message [ 1252.144980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1252.177652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1252.219958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1252.241084] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1252.264648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1252.298842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1252.340254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1258.503284] LNet: 61591:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1258.517867] LNetError: 61591:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1259.552141] LNet: Removed LNI 192.168.204.52@tcp [ 1259.554152] LNetError: 61591:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1259.582871] LNet: Removed LNI 192.168.204.52@tcp1 [ 1260.245222] Key type .llcrypt unregistered [ 1260.250953] Key type ._llcrypt unregistered [ 1279.267731] Key type ._llcrypt registered [ 1279.273611] Key type .llcrypt registered [ 1280.935640] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 15:02:13 (1789066933) [ 1282.063424] Key type .llcrypt unregistered [ 1282.073924] Key type ._llcrypt unregistered [ 1288.317086] Key type ._llcrypt registered [ 1288.322847] Key type .llcrypt registered [ 1288.697561] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1288.769150] alg: No test for adler32 (adler32-zlib) [ 1289.759476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1289.794707] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 1298.071766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.152@tcp [ 1298.909239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp -d 192.168.204.152@tcp -m GET -e remote_dropped -r 1 [ 1298.960502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp -d 192.168.204.52@tcp -m GET -e remote_dropped -r 1 [ 1298.998971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.152@tcp [ 1299.017688] LNet: There was an unexpected network error while writing to 192.168.204.152: rc = -22 [ 1299.270117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1299.302444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1299.352241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1299.404968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1299.444528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1299.492363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1299.526726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1299.561573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1299.603655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1299.625231] LNet: 62838:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1299.655742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1299.705702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1299.772235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1299.821499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1299.874092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1299.934065] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1299.978990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1300.448668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp -d 192.168.204.152@tcp -m GET -e remote_error -r 1 [ 1300.499461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp -d 192.168.204.52@tcp -m GET -e remote_error -r 1 [ 1300.534466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.152@tcp [ 1300.551577] LNet: There was an unexpected network error while writing to 192.168.204.152: rc = -22 [ 1300.773746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1300.797330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1300.825887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1300.850956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1300.879707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1300.906422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1300.943444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1300.979889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1301.029842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1301.048221] LNet: 62949:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1301.074873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1301.115676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1301.169347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1301.214951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1301.263589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1301.305590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1301.356957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1301.908956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp -d 192.168.204.152@tcp -m GET -e remote_timeout -r 1 [ 1301.945840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp -d 192.168.204.52@tcp -m GET -e remote_timeout -r 1 [ 1301.980524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.152@tcp [ 1302.000795] LNet: There was an unexpected network error while writing to 192.168.204.152: rc = -22 [ 1302.308422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1302.366106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1302.406939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1302.444548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1302.474953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1302.510845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1302.545262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1302.595550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1302.652514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1302.691514] LNet: 63061:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1302.717841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1302.755594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1302.798168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1302.846151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1302.884067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1302.932679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1302.972251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1303.320973] LNet: 63102:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1303.329223] LNetError: 63102:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1303.344741] LNet: Removed LNI 192.168.204.52@tcp [ 1303.639568] Key type .llcrypt unregistered [ 1303.644731] Key type ._llcrypt unregistered [ 1322.080388] Key type ._llcrypt registered [ 1322.084042] Key type .llcrypt registered [ 1323.125757] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 15:02:56 (1789066976) [ 1324.040605] Key type .llcrypt unregistered [ 1324.044031] Key type ._llcrypt unregistered [ 1329.475799] Key type ._llcrypt registered [ 1329.479060] Key type .llcrypt registered [ 1329.636614] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1329.655720] alg: No test for adler32 (adler32-zlib) [ 1330.548156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1330.578699] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 1337.856707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.152@tcp [ 1346.410521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1346.452660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1346.481838] LNet: Added LNI 192.168.204.52@tcp1 [8/256/0/180] [ 1347.848542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp -d 192.168.204.152@tcp -m GET -e remote_dropped -r 1 [ 1347.890782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp -d 192.168.204.152@tcp1 -m GET -e remote_dropped -r 1 [ 1347.921547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp -d 192.168.204.52@tcp -m GET -e remote_dropped -r 1 [ 1347.962388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp -d 192.168.204.52@tcp1 -m GET -e remote_dropped -r 1 [ 1347.994500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp1 -d 192.168.204.152@tcp -m GET -e remote_dropped -r 1 [ 1348.037544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp1 -d 192.168.204.152@tcp1 -m GET -e remote_dropped -r 1 [ 1348.078804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp1 -d 192.168.204.52@tcp -m GET -e remote_dropped -r 1 [ 1348.135639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp1 -d 192.168.204.52@tcp1 -m GET -e remote_dropped -r 1 [ 1348.181195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.152@tcp [ 1348.201504] LNet: There was an unexpected network error while writing to 192.168.204.152: rc = -22 [ 1348.214252] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.152@tcp [ 1348.440716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1348.469867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1348.494612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1348.528899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1348.573246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1348.608113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1348.657228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1348.695520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1348.752381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1348.768488] LNet: 64566:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1348.790185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1348.820919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1348.857149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1348.878807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1348.911452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1348.964306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1349.011001] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1349.059654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1349.759274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp -d 192.168.204.152@tcp -m GET -e remote_error -r 1 [ 1349.810878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp -d 192.168.204.152@tcp1 -m GET -e remote_error -r 1 [ 1349.866055] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp -d 192.168.204.52@tcp -m GET -e remote_error -r 1 [ 1349.930380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp -d 192.168.204.52@tcp1 -m GET -e remote_error -r 1 [ 1349.996261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp1 -d 192.168.204.152@tcp -m GET -e remote_error -r 1 [ 1350.086901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp1 -d 192.168.204.152@tcp1 -m GET -e remote_error -r 1 [ 1350.124627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp1 -d 192.168.204.52@tcp -m GET -e remote_error -r 1 [ 1350.165659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp1 -d 192.168.204.52@tcp1 -m GET -e remote_error -r 1 [ 1350.218406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.152@tcp [ 1350.234594] LNet: There was an unexpected network error while writing to 192.168.204.152: rc = -22 [ 1350.241727] LNet: Skipped 2 previous similar messages [ 1350.478847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1350.505426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1350.538288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1350.564232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1350.591076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1350.617441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1350.643825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1350.675877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1350.714320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1350.744227] LNet: 64691:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1350.777293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1350.822278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1350.870808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1350.904113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1350.949819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1350.995184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1351.036473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1351.077647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1351.626793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp -d 192.168.204.152@tcp -m GET -e remote_timeout -r 1 [ 1351.658715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp -d 192.168.204.152@tcp1 -m GET -e remote_timeout -r 1 [ 1351.696733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp -d 192.168.204.52@tcp -m GET -e remote_timeout -r 1 [ 1351.778770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp -d 192.168.204.52@tcp1 -m GET -e remote_timeout -r 1 [ 1351.835704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp1 -d 192.168.204.152@tcp -m GET -e remote_timeout -r 1 [ 1351.907279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp1 -d 192.168.204.152@tcp1 -m GET -e remote_timeout -r 1 [ 1351.950104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp1 -d 192.168.204.52@tcp -m GET -e remote_timeout -r 1 [ 1352.020363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp1 -d 192.168.204.52@tcp1 -m GET -e remote_timeout -r 1 [ 1352.076505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.152@tcp [ 1352.113427] LNet: There was an unexpected network error while writing to 192.168.204.152: rc = -22 [ 1352.401320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1352.426269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1352.473560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1352.548583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1352.600437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1352.656201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1352.694318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1352.742302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1352.795918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1352.832254] LNet: 64817:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1352.865276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1352.918648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1352.966020] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1353.018175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1353.061085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1353.105860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1353.160216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1353.188409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1358.605588] LNet: 64895:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1358.620648] LNetError: 64895:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1358.629825] LNet: Removed LNI 192.168.204.52@tcp [ 1358.669827] LNet: Removed LNI 192.168.204.52@tcp1 [ 1359.250719] Key type .llcrypt unregistered [ 1359.254603] Key type ._llcrypt unregistered [ 1375.792163] Key type ._llcrypt registered [ 1375.795164] Key type .llcrypt registered [ 1377.224825] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 15:03:49 (1789067029) [ 1379.127150] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 1380.767535] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 15:03:53 (1789067033) [ 1381.928470] Key type .llcrypt unregistered [ 1381.936946] Key type ._llcrypt unregistered [ 1388.854903] Key type ._llcrypt registered [ 1388.858215] Key type .llcrypt registered [ 1389.048183] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1389.086166] alg: No test for adler32 (adler32-zlib) [ 1390.018706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1390.068540] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 1399.161151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.152@tcp [ 1400.000864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp -d 192.168.204.152@tcp -m GET -e network_timeout -r 1 [ 1400.057588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp -d 192.168.204.52@tcp -m GET -e network_timeout -r 1 [ 1400.100883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.152@tcp [ 1400.124339] LNet: There was an unexpected network error while writing to 192.168.204.152: rc = -22 [ 1400.352183] LNet: 1 local NIs in recovery (showing 1): 192.168.204.52@tcp [ 1400.373933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1400.403199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1400.445897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1400.480077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1400.517656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1400.550444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1400.582415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1400.622380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1400.662608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1400.691944] LNet: 66437:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1400.722812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1400.759393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1400.792864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1400.830899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1400.858366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1400.901428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1401.221663] LNet: 66477:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1401.230413] LNetError: 66477:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1401.251225] LNet: Removed LNI 192.168.204.52@tcp [ 1401.640571] Key type .llcrypt unregistered [ 1401.643863] Key type ._llcrypt unregistered [ 1420.802895] Key type ._llcrypt registered [ 1420.804896] Key type .llcrypt registered [ 1420.989280] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1421.003887] alg: No test for adler32 (adler32-zlib) [ 1421.938830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1421.980891] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 1430.537745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.152@tcp [ 1441.318967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1441.373847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1441.400096] LNet: Added LNI 192.168.204.52@tcp1 [8/256/0/180] [ 1442.789952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp -d 192.168.204.152@tcp -m GET -e network_timeout -r 1 [ 1442.848574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp -d 192.168.204.152@tcp1 -m GET -e network_timeout -r 1 [ 1442.918582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp -d 192.168.204.52@tcp -m GET -e network_timeout -r 1 [ 1442.985820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp -d 192.168.204.52@tcp1 -m GET -e network_timeout -r 1 [ 1443.051594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp1 -d 192.168.204.152@tcp -m GET -e network_timeout -r 1 [ 1443.093462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp1 -d 192.168.204.152@tcp1 -m GET -e network_timeout -r 1 [ 1443.119543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp1 -d 192.168.204.52@tcp -m GET -e network_timeout -r 1 [ 1443.149741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp1 -d 192.168.204.52@tcp1 -m GET -e network_timeout -r 1 [ 1443.181118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.152@tcp [ 1443.196924] LNet: There was an unexpected network error while writing to 192.168.204.152: rc = -22 [ 1443.410835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1443.444861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1443.487888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1443.488330] LNet: 1 local NIs in recovery (showing 1): 192.168.204.52@tcp [ 1443.502448] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.152@tcp [ 1443.515752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1443.570261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1443.600374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1443.633358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1443.663292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1443.688566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1443.702829] LNet: 67353:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1443.720446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1443.745872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1443.771296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1443.792809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1443.829686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1443.879495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1448.664092] LNet: There was an unexpected network error while writing to 192.168.204.152: rc = -22 [ 1449.107408] LNet: 67427:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1449.120842] LNetError: 67427:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1449.138369] LNet: Removed LNI 192.168.204.52@tcp [ 1449.167685] LNet: Removed LNI 192.168.204.52@tcp1 [ 1449.863546] Key type .llcrypt unregistered [ 1449.870899] Key type ._llcrypt unregistered [ 1467.820645] Key type ._llcrypt registered [ 1467.825346] Key type .llcrypt registered [ 1469.294967] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 15:05:21 (1789067121) [ 1469.843308] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1469.881802] alg: No test for adler32 (adler32-zlib) [ 1470.887988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1470.960598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1470.992164] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 1471.818189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1471.836535] LNet: Added LNI 192.168.204.52@tcp1 [8/256/0/180] [ 1472.749471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.52@tcp [ 1472.823369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1472.881284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1472.946081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1472.993552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1473.032932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1473.093133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.204.52@tcp [ 1473.110615] LNetError: 68523:0:(api-ni.c:4478:lnet_ni_set_healthv()) manually adding local NI 192.168.204.52@tcp to recovery [ 1474.019804] LNet: 1 local NIs in recovery (showing 1): 192.168.204.52@tcp [ 1474.023896] LNet: There was an unexpected network error while writing to 192.168.204.52: rc = -22 [ 1476.071374] LNet: There was an unexpected network error while writing to 192.168.204.52: rc = -22 [ 1476.692654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1480.161698] LNet: There was an unexpected network error while writing to 192.168.204.52: rc = -22 [ 1480.418955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1480.463643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1480.529313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1480.549670] LNet: 68630:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1480.556848] LNetError: 68630:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1480.566269] LNet: Removed LNI 192.168.204.52@tcp [ 1480.574977] LNet: Removed LNI 192.168.204.52@tcp1 [ 1480.600392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1480.664760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1480.699698] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 1481.558716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1481.575682] LNet: Added LNI 192.168.204.52@tcp1 [8/256/0/180] [ 1482.475297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.52@tcp [ 1482.528961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1482.586635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1482.632978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1482.668376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1482.703919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1482.743627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1482.780620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.204.52@tcp [ 1482.802434] LNetError: 68903:0:(api-ni.c:4478:lnet_ni_set_healthv()) manually adding local NI 192.168.204.52@tcp to recovery [ 1483.680370] LNet: There was an unexpected network error while writing to 192.168.204.52: rc = -22 [ 1486.409943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1489.825459] LNet: There was an unexpected network error while writing to 192.168.204.52: rc = -22 [ 1489.845626] LNet: Skipped 1 previous similar message [ 1494.526492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1494.583317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1494.656897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1501.942450] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 15:05:54 (1789067154) [ 1502.152220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1502.183870] LNet: 69619:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1502.203282] LNetError: 69619:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1502.211365] LNetError: 69619:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1502.222955] LNet: Removed LNI 192.168.204.52@tcp [ 1502.256124] LNet: Removed LNI 192.168.204.52@tcp1 [ 1502.295330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1502.381788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1502.421889] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 1503.482979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1503.519735] LNet: Added LNI 192.168.204.52@tcp1 [8/256/0/180] [ 1504.618929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.52@tcp [ 1504.696640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1504.729871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1504.766167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1504.790526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1504.823027] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1504.847861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.204.52@tcp --health 0 [ 1504.938837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1505.376315] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.52@tcp [ 1505.376371] LNet: There was an unexpected network error while writing to 192.168.204.52: rc = -22 [ 1505.405228] LNet: Skipped 1 previous similar message [ 1520.582736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1520.732959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1520.767857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1520.798542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.204.52@tcp --health 500 [ 1520.939611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1527.723973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1527.805517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1527.847313] LNet: 70113:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1527.859897] LNetError: 70113:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1527.866519] LNetError: 70113:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1527.878107] LNet: Removed LNI 192.168.204.52@tcp [ 1527.895949] LNet: Removed LNI 192.168.204.52@tcp1 [ 1527.944726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1528.054749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1528.111815] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 1528.947389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1528.965308] LNet: Added LNI 192.168.204.52@tcp1 [8/256/0/180] [ 1529.992541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.52@tcp [ 1530.043485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1530.130707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1530.175714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1530.225683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1530.259092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1530.288429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1530.317841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.204.52@tcp --health 0 [ 1531.108414] LNet: There was an unexpected network error while writing to 192.168.204.52: rc = -22 [ 1531.119868] LNet: Skipped 3 previous similar messages [ 1531.693532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1542.183409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1542.218413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1542.250292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1550.680229] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 15:06:43 (1789067203) [ 1551.328555] LNet: 71160:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1551.338283] LNetError: 71160:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1551.344668] LNetError: 71160:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1551.358150] LNet: Removed LNI 192.168.204.52@tcp [ 1551.375492] LNet: Removed LNI 192.168.204.52@tcp1 [ 1552.092910] Key type .llcrypt unregistered [ 1552.101063] Key type ._llcrypt unregistered [ 1559.278561] Key type ._llcrypt registered [ 1559.281504] Key type .llcrypt registered [ 1559.553511] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1559.593130] alg: No test for adler32 (adler32-zlib) [ 1560.621416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1560.644134] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 1569.865348] Lustre: DEBUG MARKER: Initial discovery [ 1570.095179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.204.152@tcp [ 1572.909742] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1573.115291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1573.165290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.204.152@tcp [ 1573.198716] LNet: There was an unexpected network error while writing to 192.168.204.152: rc = -22 [ 1573.220706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1573.250577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.204.152@tcp -r 1 -m GET -l 3 [ 1573.283694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.204.152@tcp -r 1 -m GET -e local_error [ 1573.317771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.204.152@tcp -r 1 -m PUT -l 6 [ 1573.984486] LNet: 1 local NIs in recovery (showing 1): 192.168.204.52@tcp [ 1575.029242] Lustre: DEBUG MARKER: Force oleg452-client.virtnet to discover 192.168.204.152@tcp (in background) [ 1575.232703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.204.152@tcp [ 1576.703845] Lustre: DEBUG MARKER: Force oleg452-server to discover 192.168.204.52@tcp [ 1578.336511] LNet: There was an unexpected network error while writing to 192.168.204.152: rc = -22 [ 1579.529459] Lustre: DEBUG MARKER: Wait for 71928 [ 1581.162559] Lustre: DEBUG MARKER: Finished wait on 71928 [ 1581.226806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay del -a [ 1581.256079] LNet: There was an unexpected network error while writing to 192.168.204.152: rc = -22 [ 1581.262781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1581.674562] LNet: 72377:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1581.683965] LNetError: 72377:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1581.710350] LNet: Removed LNI 192.168.204.52@tcp [ 1582.297436] Key type .llcrypt unregistered [ 1582.299108] Key type ._llcrypt unregistered [ 1603.003886] Key type ._llcrypt registered [ 1603.005541] Key type .llcrypt registered [ 1604.658272] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 15:07:36 (1789067256) [ 1605.297939] Key type .llcrypt unregistered [ 1605.299352] Key type ._llcrypt unregistered [ 1612.067139] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1612.342557] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1612.363289] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1612.373668] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1615.073279] Key type ._llcrypt registered [ 1615.075819] Key type .llcrypt registered [ 1615.373152] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1615.390931] alg: No test for adler32 (adler32-zlib) [ 1616.379517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1616.453879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1616.485323] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 1617.409677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1617.433230] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1627.699067] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 15:08:00 (1789067280) [ 1627.902375] LNet: 74462:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1627.919654] LNetError: 74462:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1627.929065] LNet: Removed LNI 10.1.2.3@tcp [ 1627.982936] LNet: Removed LNI 192.168.204.52@tcp [ 1628.961653] Key type .llcrypt unregistered [ 1628.963438] Key type ._llcrypt unregistered [ 1640.091745] Key type ._llcrypt registered [ 1640.097624] Key type .llcrypt registered [ 1640.465825] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1640.502415] alg: No test for adler32 (adler32-zlib) [ 1641.579261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1641.633386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1641.654830] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 1642.503282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1642.560575] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1644.041992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.204.52@tcp [ 1645.152908] LNetError: Unexpected error -22 connecting to 192.168.204.52@tcp at host 192.168.204.52:988 [ 1653.922208] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 15:08:26 (1789067306) [ 1654.227534] LNet: 75596:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1654.236300] LNetError: 75596:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1654.251119] LNet: Removed LNI 10.1.2.3@tcp [ 1654.270659] LNet: Removed LNI 192.168.204.52@tcp [ 1655.205984] Key type .llcrypt unregistered [ 1655.225669] Key type ._llcrypt unregistered [ 1662.796267] Key type ._llcrypt registered [ 1662.803904] Key type .llcrypt registered [ 1663.093447] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1663.125332] alg: No test for adler32 (adler32-zlib) [ 1664.045268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1664.111558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1664.150158] LNet: Added LNI 192.168.204.52@tcp1 [8/256/0/180] [ 1665.255427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1665.279522] LNet: Added LNI 192.168.204.52@tcp2 [8/256/0/180] [ 1666.521534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.204.52@tcp1 --nid 192.168.204.52@tcp2 [ 1673.149587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.52@tcp1 192.168.204.52@tcp2 [ 1673.187744] LNetError: 76429:0:(lib-move.c:2094:lnet_handle_find_routed_path()) Unable to send message from 192.168.204.52@tcp1 to 192.168.204.52@tcp2 - Route table may be misconfigured [ 1673.259244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.52@tcp2 192.168.204.52@tcp1 [ 1682.509790] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 15:08:55 (1789067335) [ 1682.708828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1682.740423] LNet: 77002:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1682.754468] LNetError: 77002:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1682.765221] LNet: Removed LNI 192.168.204.52@tcp1 [ 1683.818932] LNet: Removed LNI 192.168.204.52@tcp2 [ 1683.870493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1683.965766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1684.002480] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 1685.000520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1685.019389] LNet: Added LNI 192.168.204.52@tcp1 [8/256/0/180] [ 1685.987982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.52@tcp [ 1686.045409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.52@tcp 192.168.204.52@tcp [ 1686.123621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.52@tcp1 192.168.204.52@tcp1 [ 1686.233701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.204.52@tcp -d 192.168.204.52@tcp -e network_timeout [ 1686.285881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.204.52@tcp -d 192.168.204.52@tcp1 -e network_timeout [ 1686.333758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.204.52@tcp1 -d 192.168.204.52@tcp -e network_timeout [ 1686.382759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.204.52@tcp1 -d 192.168.204.52@tcp1 -e network_timeout [ 1686.434908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.52@tcp [ 1686.480821] LNet: There was an unexpected network error while writing to 192.168.204.52: rc = -22 [ 1686.522067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1686.658875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1686.801867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1687.013556] LNet: 1 local NIs in recovery (showing 1): 192.168.204.52@tcp [ 1693.737502] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 15:09:06 (1789067346) [ 1694.147816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1694.175323] LNet: 77865:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1694.182361] LNetError: 77865:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1694.189279] LNetError: 77865:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1694.201249] LNet: Removed LNI 192.168.204.52@tcp [ 1695.271301] LNet: Removed LNI 192.168.204.52@tcp1 [ 1695.330860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1695.438912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1695.791619] LNet: 77902:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1696.183547] Key type .llcrypt unregistered [ 1696.185822] Key type ._llcrypt unregistered [ 1709.942194] Key type ._llcrypt registered [ 1709.944728] Key type .llcrypt registered [ 1711.128496] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 15:09:23 (1789067363) [ 1711.790971] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1711.822139] alg: No test for adler32 (adler32-zlib) [ 1712.785973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1714.447530] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1716.398675] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 15:09:28 (1789067368) [ 1716.496645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1716.538197] LNet: 78991:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1716.559553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1716.664540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1716.722603] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 1717.552672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1717.591812] LNet: Added LNI 192.168.204.52@tcp1 [8/256/0/180] [ 1718.550556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.52@tcp [ 1718.609500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.52@tcp1 [ 1718.661602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.52@tcp1 [ 1725.230250] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 15:09:38 (1789067378) [ 1726.951837] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg452-server [ 1728.653707] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 15:09:41 (1789067381) [ 1730.652881] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg452-server [ 1732.746898] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 15:09:45 (1789067385) [ 1735.043308] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg452-server [ 1736.979919] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 15:09:49 (1789067389) [ 1739.129928] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg452-server [ 1741.075646] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 15:09:53 (1789067393) [ 1743.433746] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg452-server [ 1745.342737] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 15:09:57 (1789067397) [ 1747.195836] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg452-server [ 1748.906183] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 15:10:01 (1789067401) [ 1750.867362] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg452-server [ 1752.736990] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 15:10:05 (1789067405) [ 1754.858516] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg452-server [ 1756.969541] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 15:10:09 (1789067409) [ 1759.109799] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg452-server [ 1760.952571] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 15:10:13 (1789067413) [ 1761.329604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1761.370715] LNet: 82874:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1761.376629] LNetError: 82874:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1761.390327] LNet: Removed LNI 192.168.204.52@tcp [ 1762.408690] LNet: Removed LNI 192.168.204.52@tcp1 [ 1762.458097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1762.546521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1762.578802] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 1763.771545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1763.915771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.52@tcp [ 1764.251292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1764.275118] LNet: 83022:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1764.294517] LNetError: 83022:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1764.299280] LNetError: 83022:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1765.358860] LNet: Removed LNI 192.168.204.52@tcp [ 1765.419571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1765.491705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1765.525647] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 1766.407844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.52@tcp [ 1766.741686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1766.795116] LNet: 83167:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1766.805179] LNetError: 83167:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1766.836625] LNet: Removed LNI 192.168.204.52@tcp [ 1766.901732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1767.014580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1767.059559] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 1767.993811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1768.076296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.52@tcp [ 1769.600594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1769.673979] LNetError: 83323:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1769.703891] LNet: Removed LNI 192.168.204.52@tcp [ 1769.742771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1769.851175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1769.894490] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 1770.878962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.52@tcp [ 1772.316943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1772.348778] LNet: 83478:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1772.356092] LNet: 83478:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1773.414983] LNet: Removed LNI 192.168.204.52@tcp [ 1773.511450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1773.617758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1773.646631] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 1774.349041] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1774.431170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.52@tcp [ 1774.674830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1774.718863] LNetError: 83625:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1774.726934] LNetError: 83625:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1774.744402] LNet: Removed LNI 192.168.204.52@tcp [ 1774.775648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1774.882713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1774.926076] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 1775.685387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.52@tcp [ 1775.924623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1777.005515] LNet: Removed LNI 192.168.204.52@tcp [ 1777.046669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1777.115773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1777.161315] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 1777.813242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1777.904669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.52@tcp [ 1778.146486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1779.252357] LNet: Removed LNI 192.168.204.52@tcp [ 1779.278222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1779.392845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1779.419878] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 1780.477797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.52@tcp [ 1780.724785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1780.775913] LNet: 84063:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1780.785752] LNet: 84063:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 1781.870412] LNet: Removed LNI 192.168.204.52@tcp [ 1781.903918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1781.997486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1782.023725] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 1782.878880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1782.992760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.52@tcp [ 1783.326827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1783.366978] LNetError: 84210:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1783.374387] LNetError: 84210:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 1784.424377] LNet: Removed LNI 192.168.204.52@tcp [ 1784.455423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1784.560390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1784.596995] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 1785.465888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.52@tcp [ 1786.921706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1788.009655] LNet: Removed LNI 192.168.204.52@tcp [ 1788.056589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1788.118567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1788.147563] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 1788.906639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1789.012831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.52@tcp [ 1789.318651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1790.377098] LNet: Removed LNI 192.168.204.52@tcp [ 1790.397348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1790.462481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1790.493218] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 1791.342681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.52@tcp [ 1791.695695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1792.743933] LNet: Removed LNI 192.168.204.52@tcp [ 1792.771715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1792.882921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1792.920608] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 1793.782655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1793.845140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1793.897050] LNet: Removed LNI 192.168.204.52@tcp [ 1793.917377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1794.016933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1794.071337] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 1794.836093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1794.964403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1794.992217] LNet: Removed LNI 192.168.204.52@tcp [ 1795.016938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1795.083617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 128 [ 1795.214842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1795.253978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1795.323952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 1000000000000000000000000 [ 1795.431697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1795.469283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1795.544740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer foo [ 1795.633229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1795.683625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1795.744992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 8 [ 1795.777353] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 1796.639936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1796.685023] LNet: Removed LNI 192.168.204.52@tcp [ 1796.715667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1796.779366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1804.645874] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 15:10:57 (1789067457) [ 1804.949916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1804.996639] LNet: 85679:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1805.002510] LNet: 85679:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 10 previous similar messages [ 1805.029206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1805.115749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1805.162264] LNet: Added LNI 192.168.204.52@tcp231 [8/256/0/180] [ 1806.648268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1806.681388] LNetError: 85819:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1806.692202] LNetError: 85819:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 6 previous similar messages [ 1806.716707] LNet: Removed LNI 192.168.204.52@tcp231 [ 1806.736618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1806.806779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1806.849617] LNet: Added LNI 192.168.204.52@tcp231 [8/256/0/180] [ 1806.854440] LNet: 85827:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1807.405249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1807.460796] LNet: Removed LNI 192.168.204.52@tcp231 [ 1807.477367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1807.501715] LNet: Added LNI 192.168.204.52@tcp231 [8/256/0/180] [ 1808.903944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1808.944639] LNet: Removed LNI 192.168.204.52@tcp231 [ 1808.972961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1809.106815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1809.158597] LNet: Added LNI 192.168.204.52@tcp231 [8/256/0/0] [ 1809.167524] LNet: 85979:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1809.992140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1810.044637] LNet: Removed LNI 192.168.204.52@tcp231 [ 1810.073573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1810.114667] LNet: Added LNI 192.168.204.52@tcp231 [8/256/0/0] [ 1811.564455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1811.601300] LNet: Removed LNI 192.168.204.52@tcp231 [ 1811.642157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1811.765285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1811.792691] LNet: Added LNI 192.168.204.52@tcp231 [8/256/0/60] [ 1811.802689] LNet: 86131:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1812.324085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1812.370634] LNet: Removed LNI 192.168.204.52@tcp231 [ 1812.386096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1812.415973] LNet: Added LNI 192.168.204.52@tcp231 [8/256/0/60] [ 1813.859951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1813.924630] LNet: Removed LNI 192.168.204.52@tcp231 [ 1813.952702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1814.085975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1814.127687] LNet: Added LNI 192.168.204.52@tcp231 [8/256/0/180] [ 1814.139401] LNet: 86282:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1814.633080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1814.665230] LNet: Removed LNI 192.168.204.52@tcp231 [ 1814.686739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1814.729669] LNet: Added LNI 192.168.204.52@tcp231 [8/256/0/180] [ 1823.534949] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 15:11:16 (1789067476) [ 1823.851872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1823.888594] LNet: Removed LNI 192.168.204.52@tcp231 [ 1823.917841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1823.976920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1824.036525] LNet: Added LNI 192.168.204.52@tcp232 [8/256/0/180] [ 1824.820918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1824.865398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1824.898971] LNet: Removed LNI 192.168.204.52@tcp232 [ 1824.929462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1824.961261] LNet: Added LNI 192.168.204.52@tcp232 [8/256/0/180] [ 1825.811148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1825.848785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1825.887963] LNet: Removed LNI 192.168.204.52@tcp232 [ 1825.907154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1825.951938] LNet: Added LNI 192.168.204.52@tcp232 [8/256/0/180] [ 1826.785676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1826.830194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1826.860332] LNet: Removed LNI 192.168.204.52@tcp232 [ 1826.889527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1826.917873] LNet: Added LNI 192.168.204.52@tcp232 [8/256/0/180] [ 1827.713269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1827.748733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1827.813481] LNet: Removed LNI 192.168.204.52@tcp232 [ 1827.836251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1834.793976] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 15:11:27 (1789067487) [ 1836.014159] Key type .llcrypt unregistered [ 1836.026227] Key type ._llcrypt unregistered [ 1843.420314] Key type ._llcrypt registered [ 1843.423128] Key type .llcrypt registered [ 1843.692461] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1843.727622] alg: No test for adler32 (adler32-zlib) [ 1844.698599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1844.743662] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 1854.077921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.152@tcp [ 1865.175689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1865.207705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1865.240559] LNet: Added LNI 192.168.204.52@tcp1 [8/256/0/180] [ 1866.860244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp -d 192.168.204.152@tcp -m GET -e local_interrupt -r 2 [ 1866.922158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp -d 192.168.204.152@tcp1 -m GET -e local_interrupt -r 2 [ 1866.957439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp -d 192.168.204.52@tcp -m GET -e local_interrupt -r 2 [ 1866.987314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp -d 192.168.204.52@tcp1 -m GET -e local_interrupt -r 2 [ 1867.032924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp1 -d 192.168.204.152@tcp -m GET -e local_interrupt -r 2 [ 1867.102942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp1 -d 192.168.204.152@tcp1 -m GET -e local_interrupt -r 2 [ 1867.136201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp1 -d 192.168.204.52@tcp -m GET -e local_interrupt -r 2 [ 1867.200482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp1 -d 192.168.204.52@tcp1 -m GET -e local_interrupt -r 2 [ 1867.249857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.52@tcp 192.168.204.152@tcp [ 1867.271957] LNet: There was an unexpected network error while writing to 192.168.204.152: rc = -22 [ 1867.278657] LNet: 1 local NIs in recovery (showing 1): 192.168.204.52@tcp [ 1867.601724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1867.682566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1867.755887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1867.852516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1867.927746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1867.996972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1868.076940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1868.136932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1868.176630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1868.191584] LNet: 88781:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1868.232229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1868.278407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1868.333527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1868.402575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1868.464738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1868.495519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1869.072956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp -d 192.168.204.152@tcp -m GET -e local_dropped -r 2 [ 1869.134684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp -d 192.168.204.152@tcp1 -m GET -e local_dropped -r 2 [ 1869.188880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp -d 192.168.204.52@tcp -m GET -e local_dropped -r 2 [ 1869.240817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp -d 192.168.204.52@tcp1 -m GET -e local_dropped -r 2 [ 1869.307503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp1 -d 192.168.204.152@tcp -m GET -e local_dropped -r 2 [ 1869.364575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp1 -d 192.168.204.152@tcp1 -m GET -e local_dropped -r 2 [ 1869.456514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp1 -d 192.168.204.52@tcp -m GET -e local_dropped -r 2 [ 1869.539233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp1 -d 192.168.204.52@tcp1 -m GET -e local_dropped -r 2 [ 1869.600895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.52@tcp 192.168.204.152@tcp [ 1869.628369] LNet: There was an unexpected network error while writing to 192.168.204.152: rc = -22 [ 1870.006968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1870.052151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1870.098725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1870.131790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1870.166609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1870.205764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1870.252410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1870.305643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1870.384836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1870.407055] LNet: 88904:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1870.436137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1870.477516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1870.524244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1870.566850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1870.635440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1870.673846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1871.261338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp -d 192.168.204.152@tcp -m GET -e local_aborted -r 2 [ 1871.299829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp -d 192.168.204.152@tcp1 -m GET -e local_aborted -r 2 [ 1871.352657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp -d 192.168.204.52@tcp -m GET -e local_aborted -r 2 [ 1871.394356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp -d 192.168.204.52@tcp1 -m GET -e local_aborted -r 2 [ 1871.452394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp1 -d 192.168.204.152@tcp -m GET -e local_aborted -r 2 [ 1871.501380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp1 -d 192.168.204.152@tcp1 -m GET -e local_aborted -r 2 [ 1871.538423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp1 -d 192.168.204.52@tcp -m GET -e local_aborted -r 2 [ 1871.597623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp1 -d 192.168.204.52@tcp1 -m GET -e local_aborted -r 2 [ 1871.669646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.52@tcp 192.168.204.152@tcp [ 1871.694717] LNet: There was an unexpected network error while writing to 192.168.204.152: rc = -22 [ 1872.004776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1872.045312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1872.111066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1872.165665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1872.279111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1872.343500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1872.387940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1872.445215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1872.494305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1872.530931] LNet: 89026:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1872.570316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1872.641688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1872.690656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1872.734831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1872.788431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1872.882778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1873.688197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp -d 192.168.204.152@tcp -m GET -e local_no_route -r 2 [ 1873.765294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp -d 192.168.204.152@tcp1 -m GET -e local_no_route -r 2 [ 1873.809170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp -d 192.168.204.52@tcp -m GET -e local_no_route -r 2 [ 1873.888105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp -d 192.168.204.52@tcp1 -m GET -e local_no_route -r 2 [ 1873.987195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp1 -d 192.168.204.152@tcp -m GET -e local_no_route -r 2 [ 1874.067077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp1 -d 192.168.204.152@tcp1 -m GET -e local_no_route -r 2 [ 1874.123162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp1 -d 192.168.204.52@tcp -m GET -e local_no_route -r 2 [ 1874.204890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp1 -d 192.168.204.52@tcp1 -m GET -e local_no_route -r 2 [ 1874.320972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.52@tcp 192.168.204.152@tcp [ 1874.372254] LNet: There was an unexpected network error while writing to 192.168.204.152: rc = -22 [ 1874.707912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1874.761976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1874.819039] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1874.877978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1874.934837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1874.978871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1875.023602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1875.068744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1875.123650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1875.156473] LNet: 89149:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1875.188298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1875.235158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1875.288253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1875.317115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1875.370931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1875.409234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1876.169642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp -d 192.168.204.152@tcp -m GET -e local_timeout -r 2 [ 1876.247526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp -d 192.168.204.152@tcp1 -m GET -e local_timeout -r 2 [ 1876.326779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp -d 192.168.204.52@tcp -m GET -e local_timeout -r 2 [ 1876.389773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp -d 192.168.204.52@tcp1 -m GET -e local_timeout -r 2 [ 1876.440709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp1 -d 192.168.204.152@tcp -m GET -e local_timeout -r 2 [ 1876.509112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp1 -d 192.168.204.152@tcp1 -m GET -e local_timeout -r 2 [ 1876.582499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp1 -d 192.168.204.52@tcp -m GET -e local_timeout -r 2 [ 1876.656699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp1 -d 192.168.204.52@tcp1 -m GET -e local_timeout -r 2 [ 1876.733208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.52@tcp 192.168.204.152@tcp [ 1877.052420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1877.091814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1877.125619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1877.162536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1877.199429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1877.247116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1877.299947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1877.360748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1877.412975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1877.462992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1877.531600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1877.589450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1877.640930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1877.682906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1877.739388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1878.419554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp -d 192.168.204.152@tcp -m GET -e remote_dropped -r 2 [ 1878.464759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp -d 192.168.204.152@tcp1 -m GET -e remote_dropped -r 2 [ 1878.516652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp -d 192.168.204.52@tcp -m GET -e remote_dropped -r 2 [ 1878.560423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp -d 192.168.204.52@tcp1 -m GET -e remote_dropped -r 2 [ 1878.622424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp1 -d 192.168.204.152@tcp -m GET -e remote_dropped -r 2 [ 1878.658981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp1 -d 192.168.204.152@tcp1 -m GET -e remote_dropped -r 2 [ 1878.723597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.52@tcp1 -d 192.168.204.52@tcp -m GET -e remote_dropped -r 2 [ 1878.797439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.52@tcp1 -d 192.168.204.52@tcp1 -m GET -e remote_dropped -r 2 [ 1878.862804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.52@tcp 192.168.204.152@tcp [ 1878.884799] LNet: There was an unexpected network error while writing to 192.168.204.152: rc = -22 [ 1878.888842] LNet: Skipped 1 previous similar message [ 1878.892788] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.152@tcp [ 1879.182292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1879.213136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1879.243280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1879.292475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1879.363478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1879.390212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1879.439948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1879.479615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1879.519064] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1879.549147] LNet: 89392:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1879.564729] LNet: 89392:0:(api-ni.c:301:recovery_interval_set()) Skipped 1 previous similar message [ 1879.599423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1879.659046] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1879.699252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1879.743392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1879.770461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1879.805280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1885.817385] LNet: 89467:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1885.836043] LNetError: 89467:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1885.850610] LNet: Removed LNI 192.168.204.52@tcp [ 1885.878204] LNet: Removed LNI 192.168.204.52@tcp1 [ 1886.463994] Key type .llcrypt unregistered [ 1886.466345] Key type ._llcrypt unregistered [ 1906.659572] Key type ._llcrypt registered [ 1906.662961] Key type .llcrypt registered [ 1908.365830] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 15:12:40 (1789067560) [ 1908.951539] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1908.997874] alg: No test for adler32 (adler32-zlib) [ 1909.948622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1910.057083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1910.075068] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 1910.986592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1911.037681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.204.52@tcp [ 1911.056939] LNetError: 90444:0:(api-ni.c:4478:lnet_ni_set_healthv()) manually adding local NI 192.168.204.52@tcp to recovery [ 1911.079221] LNet: 1 local NIs in recovery (showing 1): 192.168.204.52@tcp [ 1911.162449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.204.52@tcp [ 1911.318897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1918.947867] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 15:12:51 (1789067571) [ 1919.097264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1919.129105] LNet: 91026:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1919.137164] LNetError: 91026:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1919.159982] LNet: Removed LNI 192.168.204.52@tcp [ 1919.188218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1919.283309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1919.304960] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 1920.363483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1920.388247] LNet: Added LNI 192.168.204.52@tcp1 [8/256/0/180] [ 1921.620327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.204.52@tcp [ 1921.648298] LNetError: 91300:0:(api-ni.c:4478:lnet_ni_set_healthv()) manually adding local NI 192.168.204.52@tcp to recovery [ 1922.347897] LNet: There was an unexpected network error while writing to 192.168.204.52: rc = -22 [ 1923.023273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.204.52@tcp [ 1933.678531] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 15:13:05 (1789067585) [ 1936.144555] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg452-server [ 1938.356363] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 15:13:10 (1789067590) [ 1940.742578] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg452-server [ 1942.841674] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 15:13:15 (1789067595) [ 1945.000494] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg452-server [ 1946.974038] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 15:13:19 (1789067599) [ 1947.223940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1947.258871] LNet: 92931:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1947.275671] LNetError: 92931:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1948.323641] LNet: Removed LNI 192.168.204.52@tcp [ 1948.331610] LNetError: 92931:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1948.361967] LNet: Removed LNI 192.168.204.52@tcp1 [ 1948.400596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1948.521164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1948.561366] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 1949.820181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1949.900816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1949.975766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 1950.138725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1950.195836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 1950.305357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1957.974305] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 15:13:30 (1789067610) [ 1958.197660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1958.236299] LNet: 93658:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1958.251192] LNetError: 93658:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1958.282137] LNet: Removed LNI 192.168.204.52@tcp [ 1958.331301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1958.408513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1958.450129] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 1966.375596] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 15:13:38 (1789067618) [ 1966.517117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1966.559980] LNet: 94360:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1966.568717] LNetError: 94360:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1966.601768] LNet: Removed LNI 192.168.204.52@tcp [ 1966.658787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1966.757851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1966.804815] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 1975.968683] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 15:13:48 (1789067628) [ 1977.626695] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 1980.098807] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 15:13:52 (1789067632) [ 1980.555903] LNet: 95402:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1980.564089] LNetError: 95402:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1980.574335] LNet: Removed LNI 192.168.204.52@tcp [ 1981.263198] Key type .llcrypt unregistered [ 1981.265638] Key type ._llcrypt unregistered [ 1988.758305] Key type ._llcrypt registered [ 1988.762663] Key type .llcrypt registered [ 1989.097946] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1989.154606] alg: No test for adler32 (adler32-zlib) [ 1990.068076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1990.100752] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 1998.987462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.152@tcp [ 2004.403565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.204.152@tcp [ 2004.653404] LNet: 95818:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2004.662931] LNetError: 95818:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2004.681275] LNet: Removed LNI 192.168.204.52@tcp [ 2005.208520] Key type .llcrypt unregistered [ 2005.213080] Key type ._llcrypt unregistered [ 2018.505732] Key type ._llcrypt registered [ 2018.510153] Key type .llcrypt registered [ 2019.935036] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 2021.845190] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 2023.868471] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 15:14:36 (1789067676) [ 2025.466149] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 [ 2027.441265] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 15:14:40 (1789067680) [ 2029.391678] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg452-server [ 2031.512891] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 15:14:44 (1789067684) [ 2033.501941] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg452-server [ 2035.422578] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 15:14:48 (1789067688) [ 2037.724991] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2037.752984] alg: No test for adler32 (adler32-zlib) [ 2038.976813] Lustre: Lustre: Build Version: 2.17.58_40_g706ff57 [ 2039.236812] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 2040.905247] Key type lgssc registered [ 2042.323592] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2065.760530] Key type lgssc unregistered [ 2066.123655] LNet: 98991:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2066.129284] LNetError: 98991:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2066.152268] LNet: Removed LNI 192.168.204.52@tcp [ 2066.930912] Key type .llcrypt unregistered [ 2066.937145] Key type ._llcrypt unregistered [ 2085.208682] Key type ._llcrypt registered [ 2085.216822] Key type .llcrypt registered [ 2086.790431] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 15:15:39 (1789067739) [ 2087.184114] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2087.202876] alg: No test for adler32 (adler32-zlib) [ 2088.270978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2088.360675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2088.428807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2088.514552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2088.577618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2096.747229] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 15:15:49 (1789067749) [ 2097.190188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2097.244525] LNet: 100404:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2097.303208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2097.452244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2097.481293] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 2098.650632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2098.709670] LNet: 100548:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2098.719080] LNetError: 100548:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2098.735350] LNet: Removed LNI 192.168.204.52@tcp [ 2098.776913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2098.847996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 2098.914764] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 2099.704484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2099.735873] LNet: 100576:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2099.747833] LNetError: 100576:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2099.765214] LNet: Removed LNI 192.168.204.52@tcp [ 2099.788413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2099.835320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 2099.858804] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 2108.393967] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 15:16:00 (1789067760) [ 2108.504548] LNet: 101158:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2108.523562] LNetError: 101158:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2108.537932] LNet: Removed LNI 192.168.204.52@tcp [ 2109.398967] Key type .llcrypt unregistered [ 2109.401479] Key type ._llcrypt unregistered [ 2117.818662] Key type ._llcrypt registered [ 2117.826107] Key type .llcrypt registered [ 2118.026485] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2118.042509] alg: No test for adler32 (adler32-zlib) [ 2119.030521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2119.144398] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 2119.149861] LNet: *** cfs_fail_loc=e003, val=100*** [ 2120.108937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.52@tcp [ 2120.128194] LNet: *** cfs_fail_loc=e003, val=100*** [ 2120.131444] LNet: Skipped 2 previous similar messages [ 2120.138307] Ping reply received with mapping: [ 2120.143092] found metadata reply NI[192.168.204.52@tcp] [ 2120.146590] 34 cc a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 2120.150726] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 2120.152987] 5a 5a 5a 5a 5a 5a 5a 5a [ 2120.299613] LNet: 101562:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2120.307788] LNetError: 101562:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2121.386364] LNet: Removed LNI 192.168.204.52@tcp [ 2122.238523] Key type .llcrypt unregistered [ 2122.242790] Key type ._llcrypt unregistered [ 2135.666118] Key type ._llcrypt registered [ 2135.670405] Key type .llcrypt registered [ 2137.047944] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 15:16:29 (1789067789) [ 2137.773559] Key type .llcrypt unregistered [ 2137.777617] Key type ._llcrypt unregistered [ 2145.091645] Key type ._llcrypt registered [ 2145.102251] Key type .llcrypt registered [ 2145.520932] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2145.535974] alg: No test for adler32 (adler32-zlib) [ 2146.811405] LNetError: 102599:0:(api-ni.c:2855:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 2147.386541] Key type .llcrypt unregistered [ 2147.394976] Key type ._llcrypt unregistered [ 2153.112034] Key type ._llcrypt registered [ 2153.113632] Key type .llcrypt registered [ 2154.532734] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 15:16:47 (1789067807) [ 2155.120379] Key type .llcrypt unregistered [ 2155.122136] Key type ._llcrypt unregistered [ 2161.505680] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2161.508295] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2164.198067] Key type ._llcrypt registered [ 2164.201742] Key type .llcrypt registered [ 2164.481828] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2164.503609] alg: No test for adler32 (adler32-zlib) [ 2165.413628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2165.480110] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2175.345369] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 15:17:07 (1789067827) [ 2175.504937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2175.531472] LNet: 104370:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2175.538536] LNetError: 104370:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2175.558803] LNet: Removed LNI 10.1.2.3@tcp [ 2175.639669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2176.396612] Key type .llcrypt unregistered [ 2176.400797] Key type ._llcrypt unregistered [ 2182.875237] Key type ._llcrypt registered [ 2182.877848] Key type .llcrypt registered [ 2183.029277] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2183.042678] alg: No test for adler32 (adler32-zlib) [ 2184.068564] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 2184.164095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.52@tcp [ 2184.349696] LNet: 104659:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2184.359785] LNetError: 104659:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2185.389106] LNet: Removed LNI 192.168.204.52@tcp [ 2186.076985] Key type .llcrypt unregistered [ 2186.082591] Key type ._llcrypt unregistered [ 2198.656600] Key type ._llcrypt registered [ 2198.660433] Key type .llcrypt registered [ 2199.857706] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 15:17:32 (1789067852) [ 2200.835483] Key type .llcrypt unregistered [ 2200.839921] Key type ._llcrypt unregistered [ 2207.606632] Key type ._llcrypt registered [ 2207.609820] Key type .llcrypt registered [ 2207.920688] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2207.952703] alg: No test for adler32 (adler32-zlib) [ 2215.583174] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 15:17:48 (1789067868) [ 2215.717468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2215.773479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2215.900660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2215.926575] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 2216.895958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2216.968990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 2217.024956] LNet: Removed LNI 192.168.204.52@tcp [ 2217.396737] LNet: 106401:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2217.876657] Key type .llcrypt unregistered [ 2217.881768] Key type ._llcrypt unregistered [ 2230.243258] Key type ._llcrypt registered [ 2230.247379] Key type .llcrypt registered [ 2231.715488] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 15:18:04 (1789067884) [ 2232.158186] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2232.172784] alg: No test for adler32 (adler32-zlib) [ 2233.200498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2233.260792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2233.281094] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 2234.310740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.52@tcp [ 2234.399338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.204.52@tcp [ 2234.427830] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 2234.433208] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 2234.437877] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2234, 0, 0 [ 2234.443252] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 2234.446897] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 2234.451785] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 2234.455784] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2234, 0, 0 [ 2234.461419] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 2234.465194] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2234.469168] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 2234.471723] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2234, 0, 2284 [ 2234.478574] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 2234.481697] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2234.486346] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 2234.491539] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2234, 0, 2284 [ 2234.498629] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 2242.137618] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 15:18:14 (1789067894) [ 2242.594857] LNet: 107939:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2242.602484] LNetError: 107939:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2243.628852] LNet: Removed LNI 192.168.204.52@tcp [ 2244.320652] Key type .llcrypt unregistered [ 2244.325078] Key type ._llcrypt unregistered [ 2251.443067] Key type ._llcrypt registered [ 2251.444736] Key type .llcrypt registered [ 2251.642380] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2251.666329] alg: No test for adler32 (adler32-zlib) [ 2252.663047] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2252.710991] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 2261.837254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.152@tcp [ 2272.578307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2272.622852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2272.661669] LNet: Added LNI 192.168.204.52@tcp1 [8/256/0/180] [ 2273.804148] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2273.974765] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2273.981382] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2273.987380] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2276.417472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 2276.432784] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 2287.237753] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 15:18:59 (1789067939) [ 2287.559432] LNet: 109444:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2287.580192] LNetError: 109444:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2287.600091] LNet: Removed LNI 192.168.204.52@tcp [ 2288.672806] LNet: Removed LNI 192.168.204.52@tcp1 [ 2288.676683] LNetError: 109444:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2288.682242] LNetError: 109444:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2288.701698] LNet: Removed LNI 10.1.2.3@tcp99 [ 2289.543575] Key type .llcrypt unregistered [ 2289.548453] Key type ._llcrypt unregistered [ 2297.611799] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2297.623622] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2300.270064] Key type ._llcrypt registered [ 2300.276124] Key type .llcrypt registered [ 2300.491814] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2300.514919] alg: No test for adler32 (adler32-zlib) [ 2301.497578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2301.552277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2301.576874] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 2302.227963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2302.246282] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2304.672734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 2304.730394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2304.796418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 2304.822739] LNetError: 110186:0:(peer.c:2366:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 2304.861197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2304.918421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.52@tcp [ 2305.513765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2305.598878] LNet: 110206:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2305.606145] LNetError: 110206:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2305.614813] LNet: Removed LNI 10.1.2.3@tcp [ 2306.673678] LNet: Removed LNI 192.168.204.52@tcp [ 2307.421236] Key type .llcrypt unregistered [ 2307.428145] Key type ._llcrypt unregistered [ 2319.696966] Key type ._llcrypt registered [ 2319.700958] Key type .llcrypt registered [ 2321.071344] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 15:19:33 (1789067973) [ 2321.702869] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2321.730432] alg: No test for adler32 (adler32-zlib) [ 2322.688903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2322.765257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2322.809742] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 2323.539411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.52@tcp [ 2323.608067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg452-client@tcp [ 2331.076758] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 15:19:43 (1789067983) [ 2331.407253] LNet: 111737:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2331.418118] LNetError: 111737:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2331.436903] LNet: Removed LNI 192.168.204.52@tcp [ 2332.318554] Key type .llcrypt unregistered [ 2332.323838] Key type ._llcrypt unregistered [ 2340.575507] Key type ._llcrypt registered [ 2340.579616] Key type .llcrypt registered [ 2340.967722] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2340.996488] alg: No test for adler32 (adler32-zlib) [ 2342.013569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 2342.086236] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 2342.153468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 2349.720508] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 15:20:02 (1789068002) [ 2352.122291] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 2353.671438] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 15:20:06 (1789068006) [ 2355.493285] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 2357.718094] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 15:20:09 (1789068009) [ 2357.818701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2357.884507] LNet: 113152:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2357.898897] LNetError: 113152:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2357.923182] LNet: Removed LNI 192.168.204.52@tcp [ 2357.958330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2358.060565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2358.087752] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 2358.908348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2359.010760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2367.067703] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 15:20:19 (1789068019) [ 2367.270623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2367.300825] LNet: 113869:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2367.316517] LNetError: 113869:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2367.340107] LNet: Removed LNI 192.168.204.52@tcp [ 2367.393757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2367.462461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 2367.510162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2368.325760] Key type .llcrypt unregistered [ 2368.331715] Key type ._llcrypt unregistered [ 2379.913058] Key type ._llcrypt registered [ 2379.919698] Key type .llcrypt registered [ 2381.299708] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 15:20:33 (1789068033) [ 2381.739706] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2381.762144] alg: No test for adler32 (adler32-zlib) [ 2382.709128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2382.818830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2382.869664] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 2383.777230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2383.834294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.52@tcp [ 2391.030935] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 15:20:43 (1789068043) [ 2391.357263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2391.409048] LNet: 115422:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2391.419215] LNetError: 115422:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2391.446609] LNet: Removed LNI 192.168.204.52@tcp [ 2391.487922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2391.546930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 2391.584693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 2398.496273] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 15:20:51 (1789068051) [ 2398.900398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2398.928397] LNet: 116003:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2398.963784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2399.054208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2399.670203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2399.714452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2399.823673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 2400.338342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2400.381318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2400.481330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2400.521857] LNet: 116036:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2400.567877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2400.651322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2401.171642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2401.203472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2401.299113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 2401.816747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2401.849348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2409.192394] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 15:21:01 (1789068061) [ 2409.404622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2409.435907] LNet: 116636:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2409.454170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2409.513642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2409.548301] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 2410.269942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2410.304895] LNet: Added LNI 192.168.204.52@tcp2 [8/256/0/180] [ 2411.142987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2411.215859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.52@tcp [ 2411.474737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2411.497437] LNetError: 116916:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2411.504224] LNet: Removed LNI 192.168.204.52@tcp [ 2411.514974] LNet: Removed LNI 192.168.204.52@tcp2 [ 2411.553453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2411.617355] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 2411.624064] LNet: Added LNI 192.168.204.52@tcp2 [8/256/0/180] [ 2411.630253] LNet: 116922:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2412.366779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.52@tcp [ 2412.650612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2412.687355] LNetError: 116959:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2412.693378] LNetError: 116959:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2412.709397] LNet: Removed LNI 192.168.204.52@tcp [ 2412.730356] LNet: Removed LNI 192.168.204.52@tcp2 [ 2412.754620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2412.895379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2412.952976] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 2413.720165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2413.740976] LNet: Added LNI 192.168.204.52@tcp2 [8/256/0/180] [ 2414.610624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.52@tcp [ 2414.650645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2414.700535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.52@tcp [ 2414.996684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2415.016760] LNet: 117243:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2415.023967] LNet: 117243:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 2415.038816] LNetError: 117243:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2415.052600] LNetError: 117243:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2415.064530] LNet: Removed LNI 192.168.204.52@tcp [ 2416.104832] LNet: Removed LNI 192.168.204.52@tcp2 [ 2416.170779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2416.270328] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 2416.278065] LNet: Added LNI 192.168.204.52@tcp2 [8/256/0/180] [ 2416.289515] LNet: 117249:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2416.997601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.52@tcp [ 2417.267362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2417.295668] LNet: Removed LNI 192.168.204.52@tcp [ 2418.346402] LNet: Removed LNI 192.168.204.52@tcp2 [ 2418.395602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2418.450776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2418.488423] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 2419.191280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2419.218300] LNet: Added LNI 192.168.204.52@tcp2 [8/256/0/180] [ 2419.991287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2420.045196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.52@tcp [ 2420.269720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2420.289804] LNetError: 117566:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2420.294193] LNetError: 117566:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 2421.345293] LNet: Removed LNI 192.168.204.52@tcp [ 2421.355903] LNet: Removed LNI 192.168.204.52@tcp2 [ 2421.407057] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2421.468366] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 2421.475340] LNet: Added LNI 192.168.204.52@tcp2 [8/256/0/180] [ 2421.480140] LNet: 117572:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2422.243890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.52@tcp [ 2422.510077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2423.584882] LNet: Removed LNI 192.168.204.52@tcp [ 2423.603797] LNet: Removed LNI 192.168.204.52@tcp2 [ 2423.654944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2423.740348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2423.787315] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 2424.625626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2424.651109] LNet: Added LNI 192.168.204.52@tcp2 [8/256/0/180] [ 2425.676310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.52@tcp [ 2425.710936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2425.782070] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.52@tcp [ 2425.983332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2426.020394] LNet: 117894:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2426.027701] LNet: 117894:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 2427.104397] LNet: Removed LNI 192.168.204.52@tcp [ 2428.133671] LNet: Removed LNI 192.168.204.52@tcp2 [ 2428.182448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2428.281300] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 2428.294300] LNet: Added LNI 192.168.204.52@tcp2 [8/256/0/180] [ 2428.310648] LNet: 117901:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2429.067131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.52@tcp [ 2429.340966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2429.366922] LNetError: 117938:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2429.372098] LNetError: 117938:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 2429.378288] LNet: Removed LNI 192.168.204.52@tcp [ 2430.448727] LNet: Removed LNI 192.168.204.52@tcp2 [ 2430.477401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2430.547683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2430.610652] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 2431.245066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2431.266460] LNet: Added LNI 192.168.204.52@tcp2 [8/256/0/180] [ 2432.048302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2432.079709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2432.140971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.52@tcp [ 2432.500315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2433.568126] LNet: Removed LNI 192.168.204.52@tcp [ 2433.582600] LNet: Removed LNI 192.168.204.52@tcp2 [ 2433.627917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2433.695155] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 2433.700436] LNet: Added LNI 192.168.204.52@tcp2 [8/256/0/180] [ 2433.707888] LNet: 118242:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2434.178624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.52@tcp [ 2434.493935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2435.552623] LNet: Removed LNI 192.168.204.52@tcp [ 2435.580800] LNet: Removed LNI 192.168.204.52@tcp2 [ 2435.615723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2435.687200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2435.714774] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 2436.467615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2436.497181] LNet: Added LNI 192.168.204.52@tcp2 [8/256/0/180] [ 2437.293254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.52@tcp [ 2437.327296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2437.355744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2437.416093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.52@tcp [ 2437.818457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2438.880147] LNet: Removed LNI 192.168.204.52@tcp [ 2439.914059] LNet: Removed LNI 192.168.204.52@tcp2 [ 2439.996346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2440.074282] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 2440.088299] LNet: Added LNI 192.168.204.52@tcp2 [8/256/0/180] [ 2440.908306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.52@tcp [ 2447.852093] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 2449.415728] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 15:21:42 (1789068102) [ 2449.630589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2449.692619] LNet: 119361:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2449.701525] LNet: 119361:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 2449.717277] LNetError: 119361:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2449.733632] LNetError: 119361:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 2449.741447] LNet: Removed LNI 192.168.204.52@tcp [ 2450.801972] LNet: Removed LNI 192.168.204.52@tcp2 [ 2450.839982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2451.176364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2451.200947] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 2451.848383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2451.864243] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2452.782346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2452.803564] LNet: Removed LNI 10.1.2.3@tcp [ 2452.817304] LNet: Removed LNI 192.168.204.52@tcp [ 2452.840763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2452.916757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 2452.957047] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 2452.964796] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2453.400136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 2453.424749] LNet: Removed LNI 192.168.204.52@tcp [ 2453.447222] LNet: Removed LNI 10.1.2.3@tcp [ 2459.918090] Lustre: DEBUG MARKER: == sanity-lnet test 475: per-operation latency stats populate and reset ========================================================== 15:21:52 (1789068112) [ 2460.061437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2460.146530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2460.290753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2460.339756] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 2461.254388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2461.320832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2461.374817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.52@tcp [ 2461.428986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.52@tcp [ 2461.462325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.52@tcp [ 2461.500264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.52@tcp [ 2461.547078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.52@tcp [ 2461.590822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.52@tcp [ 2461.630356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.52@tcp [ 2461.665731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.52@tcp [ 2461.719666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.52@tcp [ 2461.784935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.52@tcp [ 2461.936597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2468.677820] Lustre: DEBUG MARKER: == sanity-lnet test 476: lnetctl set latency_stats toggles capture ========================================================== 15:22:01 (1789068121) [ 2468.905481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2470.011590] LNet: Removed LNI 192.168.204.52@tcp [ 2470.054771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2470.148318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2470.178219] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 2471.110695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 0 [ 2471.155927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2471.224485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.52@tcp [ 2471.265542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.52@tcp [ 2471.310940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.52@tcp [ 2471.347820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.52@tcp [ 2471.406078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.52@tcp [ 2471.463610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.52@tcp [ 2471.513341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.52@tcp [ 2471.548680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.52@tcp [ 2471.587461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.52@tcp [ 2471.619785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.52@tcp [ 2471.740255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2471.781567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.52@tcp [ 2471.816470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.52@tcp [ 2471.850733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.52@tcp [ 2471.878168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.52@tcp [ 2471.907197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.52@tcp [ 2471.931651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.52@tcp [ 2471.963312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.52@tcp [ 2472.007271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.52@tcp [ 2472.062680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.52@tcp [ 2472.122635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.52@tcp [ 2479.431976] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 15:22:11 (1789068131) [ 2479.527202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2480.617600] LNet: Removed LNI 192.168.204.52@tcp [ 2480.685931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2480.852836] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2481.040484] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2481.044433] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2481.052635] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2483.383108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2483.435705] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 2484.247572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2484.271624] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2486.723732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.52@tcp [ 2486.772805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 2486.852193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.204.52@tcp -m PUT -e local_timeout -r 1 [ 2486.914837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 2486.979920] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2486.986775] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2487.277330] LNetError: 121791:0:(lib-move.c:4794:lnet_parse()) 192.168.204.52@tcp, src 192.168.204.52@tcp: Dropping PUT (error -108 looking up sender) [ 2488.289765] LNet: 122250:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2488.299227] LNet: 122250:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 2488.314709] LNetError: 122250:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2488.323917] LNetError: 122250:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 2488.333841] LNet: Removed LNI 10.1.2.3@tcp [ 2489.392362] LNet: Removed LNI 192.168.204.52@tcp [ 2489.767097] Key type .llcrypt unregistered [ 2489.769191] Key type ._llcrypt unregistered [ 2504.279528] Key type ._llcrypt registered [ 2504.283343] Key type .llcrypt registered [ 2505.675263] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 15:22:38 (1789068158) [ 2506.165947] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2506.187104] alg: No test for adler32 (adler32-zlib) [ 2507.116650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2507.577538] LNet: 123101:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2508.167629] Key type .llcrypt unregistered [ 2508.174885] Key type ._llcrypt unregistered [ 2516.047517] Key type ._llcrypt registered [ 2516.050607] Key type .llcrypt registered [ 2516.300619] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2516.345261] alg: No test for adler32 (adler32-zlib) [ 2517.310103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2517.334756] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 2524.706319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.152@tcp [ 2525.007098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 2525.702698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.204.152@tcp [ 2525.991684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2526.026713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2526.063155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2526.097739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2526.139269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2526.163534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2526.201515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2526.247573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2526.288461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2526.307826] LNet: 123551:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2526.331902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2526.383780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2526.422218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2526.462740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2526.502991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2526.541707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2527.339457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2527.578129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2527.589144] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.152@tcp [ 2527.636385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2527.678292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2527.736412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2527.791867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2527.828918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2527.859238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2527.901542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2527.974569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2528.005501] LNet: 123670:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2528.047594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2528.114603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2528.149467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2528.184175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2528.230403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2528.284742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2528.811798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.204.152@tcp [ 2529.053392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2529.106731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2529.140071] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2529.167778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2529.216723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2529.255082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2529.303372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2529.353851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2529.399352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2529.442825] LNet: 123789:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2529.463950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2529.528588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2529.573799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2529.626583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2529.685152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2529.722505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2530.388188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2530.667437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2530.706992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2530.738734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2530.770269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2530.806333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2530.844316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2530.897258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2530.936069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2530.996746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2531.035829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2531.067290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2531.102512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2531.141265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2531.187374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2531.226874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2531.507487] LNet: 123955:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2531.512914] LNetError: 123955:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2531.531155] LNet: Removed LNI 192.168.204.52@tcp [ 2531.975997] Key type .llcrypt unregistered [ 2531.978090] Key type ._llcrypt unregistered [ 2549.123917] Key type ._llcrypt registered [ 2549.128464] Key type .llcrypt registered [ 2550.518488] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 15:23:23 (1789068203) [ 2550.978885] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2551.002725] alg: No test for adler32 (adler32-zlib) [ 2551.998283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2552.490740] LNet: 124841:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2553.188697] Key type .llcrypt unregistered [ 2553.191672] Key type ._llcrypt unregistered [ 2560.972144] Key type ._llcrypt registered [ 2560.980472] Key type .llcrypt registered [ 2561.225738] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2561.242181] alg: No test for adler32 (adler32-zlib) [ 2562.159361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2562.184082] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 2570.671166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.152@tcp [ 2581.597561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2581.653912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2581.690726] LNet: Added LNI 192.168.204.52@tcp1 [8/256/0/180] [ 2582.629321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 2583.222964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.204.152@tcp [ 2583.473996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2583.514120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2583.559734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2583.589636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2583.627085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2583.656876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2583.683707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2583.710394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2583.719425] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.152@tcp [ 2583.741529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2583.755737] LNet: 125494:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2583.777417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2583.828835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2583.866467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2583.907663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2583.939660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2583.989481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2584.777577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2585.080646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2585.126979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2585.168457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2585.201716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2585.249675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2585.290134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2585.313643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2585.340555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2585.361759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2585.374971] LNet: 125613:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2585.392542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2585.423603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2585.463615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2585.524951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2585.584462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2585.635465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2586.251640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.204.152@tcp [ 2586.572469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2586.640178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2586.689697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2586.728442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2586.768234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2586.817467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2586.865173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2586.919379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2586.955345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2586.971910] LNet: 125732:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2586.994670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2587.029393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2587.089246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2587.127988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2587.162357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2587.206986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2587.882900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2588.166565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2588.201470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2588.240167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2588.275380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2588.319344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2588.365411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2588.399814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2588.455445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2588.485454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2588.518992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2588.543653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2588.579597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2588.620809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2588.683793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2588.748345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2594.519651] LNet: 125933:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2594.535486] LNetError: 125933:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2594.546103] LNet: Removed LNI 192.168.204.52@tcp [ 2594.566842] LNet: Removed LNI 192.168.204.52@tcp1 [ 2595.055813] Key type .llcrypt unregistered [ 2595.061087] Key type ._llcrypt unregistered [ 2614.669774] Key type ._llcrypt registered [ 2614.677547] Key type .llcrypt registered [ 2616.542340] Lustre: DEBUG MARKER: == sanity-lnet test 525: keep reachable peer on local net delete ========================================================== 15:24:28 (1789068268) [ 2617.141544] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2617.168941] alg: No test for adler32 (adler32-zlib) [ 2618.056682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2618.164344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2618.207069] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 2619.602088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2619.626521] LNet: Added LNI 192.168.204.52@tcp2 [8/256/0/180] [ 2621.210826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.52@tcp [ 2621.319404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp [ 2622.368770] LNet: Removed LNI 192.168.204.52@tcp [ 2629.255733] Lustre: DEBUG MARKER: == sanity-lnet test 550: DD-off: keep route up when gw reachable via another local NI ========================================================== 15:24:41 (1789068281) [ 2631.434959] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_550 Need at least 2 local interfaces for multi-rail [ 2634.781074] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 15:24:46 (1789068286) [ 2644.212538] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 15:24:56 (1789068296) [ 2652.968725] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 15:25:05 (1789068305) [ 2660.598223] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 15:25:13 (1789068313) [ 2668.737536] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 15:25:21 (1789068321) [ 2677.468278] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 15:25:30 (1789068330) [ 2685.851894] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 15:25:38 (1789068338) [ 2694.286243] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 15:25:46 (1789068346) [ 2702.806737] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 15:25:55 (1789068355) [ 2710.424725] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 15:26:03 (1789068363) [ 2718.062339] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 15:26:10 (1789068370) [ 2727.618835] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 15:26:19 (1789068379) [ 2735.967072] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 15:26:28 (1789068388) [ 2742.797916] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 15:26:36 (1789068396) [ 2749.668801] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 15:26:42 (1789068402) [ 2756.534486] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 15:26:49 (1789068409) [ 2764.137301] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 15:26:56 (1789068416) [ 2771.394621] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 15:27:04 (1789068424) [ 2771.560483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2771.595248] LNet: 137628:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2771.611343] LNetError: 137628:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2771.631896] LNet: Removed LNI 192.168.204.52@tcp2 [ 2771.659446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2771.798828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2771.822115] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 2773.068673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2773.094606] LNet: 137771:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2773.104757] LNetError: 137771:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2773.119621] LNet: Removed LNI 192.168.204.52@tcp [ 2773.136670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2773.178635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2773.193066] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 2781.067220] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 15:27:13 (1789068433) [ 2788.588710] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 15:27:21 (1789068441) [ 2796.246099] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 15:27:28 (1789068448) [ 2804.412667] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 15:27:37 (1789068457) [ 2812.976960] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 15:27:45 (1789068465) [ 2821.821405] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 15:27:54 (1789068474) [ 2830.300786] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 15:28:03 (1789068483) [ 2838.410577] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 15:28:11 (1789068491) [ 2846.297795] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 15:28:19 (1789068499) [ 2854.613827] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 15:28:27 (1789068507) [ 2862.144905] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 15:28:34 (1789068514) [ 2869.316960] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 15:28:42 (1789068522) [ 2876.095302] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 2628 sec ========== 15:28:48 (1789068528) [ 2876.406743] LNet: 145240:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2876.417808] LNetError: 145240:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2876.430938] LNet: Removed LNI 192.168.204.52@tcp [ 2877.109589] Key type .llcrypt unregistered [ 2877.112550] Key type ._llcrypt unregistered [ 2890.564435] Key type ._llcrypt registered [ 2890.574807] Key type .llcrypt registered [ 2891.060866] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2891.076940] alg: No test for adler32 (adler32-zlib) [ 2892.220415] Lustre: Lustre: Build Version: 2.17.58_40_g706ff57 [ 2892.553641] LNet: Added LNI 192.168.204.52@tcp [8/256/0/180] [ 2894.284317] Key type lgssc registered [ 2895.963327] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2962.465434] Lustre: Mounted lustre-client - version 2.17.58_40_g706ff57 [ 2967.645651] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2975.097670] Lustre: Unmounted lustre-client [ 3013.561330] Key type lgssc unregistered [ 3013.945199] LNet: 148188:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3013.959469] LNetError: 148188:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3013.974506] LNet: Removed LNI 192.168.204.52@tcp [ 3014.648203] Key type .llcrypt unregistered [ 3014.651814] Key type ._llcrypt unregistered