[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 458283492 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.002398] x2apic enabled [ 0.003008] Switched APIC routing to physical x2apic. [ 0.004015] kvm-guest: setup PV IPIs [ 0.007000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007030] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008015] pid_max: default: 32768 minimum: 301 [ 0.009157] LSM: Security Framework initializing [ 0.010068] Yama: becoming mindful. [ 0.011042] SELinux: Initializing. [ 0.012069] *** VALIDATE selinux *** [ 0.020669] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.025031] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027015] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028132] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029137] *** VALIDATE tmpfs *** [ 0.031090] *** VALIDATE proc *** [ 0.032228] *** VALIDATE cgroup *** [ 0.033006] *** VALIDATE cgroup2 *** [ 0.034131] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.035167] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.036009] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.037031] Spectre V2 : User space: Vulnerable [ 0.038008] Speculative Store Bypass: Vulnerable [ 0.041490] debug: unmapping init [mem 0xffffffffa9059000-0xffffffffa9060fff] [ 0.044000] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.044741] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.045029] ... version: 2 [ 0.046014] ... bit width: 48 [ 0.047013] ... generic registers: 4 [ 0.048016] ... value mask: 0000ffffffffffff [ 0.049014] ... max period: 00007fffffffffff [ 0.050014] ... fixed-purpose events: 3 [ 0.051014] ... event mask: 000000070000000f [ 0.052381] rcu: Hierarchical SRCU implementation. [ 0.054410] smp: Bringing up secondary CPUs ... [ 0.055548] x86: Booting SMP configuration: [ 0.056026] .... node #0, CPUs: #1 #2 #3 [ 0.059552] smp: Brought up 1 node, 4 CPUs [ 0.061000] smpboot: Max logical packages: 1 [ 0.062029] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.140022] node 0 deferred pages initialised in 76ms [ 0.143133] devtmpfs: initialized [ 0.144215] x86/mm: Memory block size: 128MB [ 0.146746] gcov: version magic: 0x41383552 [ 0.148154] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.149074] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.150261] pinctrl core: initialized pinctrl subsystem [ 0.151179] [ 0.151628] ************************************************************* [ 0.152014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.153012] ** ** [ 0.154012] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.155016] ** ** [ 0.156015] ** This means that this kernel is built to expose internal ** [ 0.157015] ** IOMMU data structures, which may compromise security on ** [ 0.158012] ** your system. ** [ 0.159016] ** ** [ 0.160014] ** If you see this message and you are not debugging the ** [ 0.161015] ** kernel, report this immediately to your vendor! ** [ 0.162018] ** ** [ 0.163013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.164014] ************************************************************* [ 0.165675] NET: Registered protocol family 16 [ 0.166431] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.167060] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.168063] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.169496] cpuidle: using governor menu [ 0.172790] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.175561] PCI: Using configuration type 1 for base access [ 0.178133] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.185073] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.186046] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.188076] cryptd: max_cpu_qlen set to 1000 [ 0.190293] ACPI: Added _OSI(Module Device) [ 0.192011] ACPI: Added _OSI(Processor Device) [ 0.193012] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.194010] ACPI: Added _OSI(Processor Aggregator Device) [ 0.199107] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.203433] ACPI: Interpreter enabled [ 0.205067] ACPI: PM: (supports S0 S3 S4 S5) [ 0.206012] ACPI: Using IOAPIC for interrupt routing [ 0.208157] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.211391] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.220405] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.223044] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.226019] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.230087] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.234059] acpiphp: Slot [2] registered [ 0.235033] acpiphp: Slot [5] registered [ 0.236091] acpiphp: Slot [6] registered [ 0.238096] acpiphp: Slot [3] registered [ 0.239159] acpiphp: Slot [4] registered [ 0.240098] acpiphp: Slot [7] registered [ 0.242091] acpiphp: Slot [8] registered [ 0.243099] acpiphp: Slot [9] registered [ 0.245098] acpiphp: Slot [10] registered [ 0.246103] acpiphp: Slot [11] registered [ 0.248117] acpiphp: Slot [12] registered [ 0.249103] acpiphp: Slot [13] registered [ 0.251096] acpiphp: Slot [14] registered [ 0.252091] acpiphp: Slot [15] registered [ 0.253096] acpiphp: Slot [16] registered [ 0.255102] acpiphp: Slot [17] registered [ 0.256130] acpiphp: Slot [18] registered [ 0.258109] acpiphp: Slot [19] registered [ 0.259092] acpiphp: Slot [20] registered [ 0.261098] acpiphp: Slot [21] registered [ 0.262076] acpiphp: Slot [22] registered [ 0.263076] acpiphp: Slot [23] registered [ 0.265081] acpiphp: Slot [24] registered [ 0.266104] acpiphp: Slot [25] registered [ 0.267120] acpiphp: Slot [26] registered [ 0.269080] acpiphp: Slot [27] registered [ 0.270081] acpiphp: Slot [28] registered [ 0.271082] acpiphp: Slot [29] registered [ 0.272077] acpiphp: Slot [30] registered [ 0.274087] acpiphp: Slot [31] registered [ 0.275071] PCI host bridge to bus 0000:00 [ 0.276017] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.278018] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.280019] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.282020] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.284022] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.287031] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.288183] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.290929] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.294170] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.299963] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.303991] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.307016] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.309016] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.311014] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.313545] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.314591] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.316039] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.318846] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.322974] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.332952] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.337018] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.341844] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.365019] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.380018] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.423018] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.431543] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.440015] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.451022] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.478020] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.494955] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.497421] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.499353] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.501383] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.503195] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.508108] iommu: Default domain type: Passthrough [ 0.510460] SCSI subsystem initialized [ 0.511181] ACPI: bus type USB registered [ 0.513126] usbcore: registered new interface driver usbfs [ 0.515076] usbcore: registered new interface driver hub [ 0.517111] usbcore: registered new device driver usb [ 0.519162] pps_core: LinuxPPS API ver. 1 registered [ 0.521011] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.524065] PTP clock support registered [ 0.527048] EDAC MC: Ver: 3.0.0 [ 0.529132] PCI: Using ACPI for IRQ routing [ 0.531723] NetLabel: Initializing [ 0.532009] NetLabel: domain hash size = 128 [ 0.534011] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.536081] NetLabel: unlabeled traffic allowed by default [ 0.538154] vgaarb: loaded [ 0.539277] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.541016] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.550367] clocksource: Switched to clocksource kvm-clock [ 0.659292] VFS: Disk quotas dquot_6.6.0 [ 0.660950] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.663489] *** VALIDATE ramfs *** [ 0.664628] *** VALIDATE hugetlbfs *** [ 0.666230] pnp: PnP ACPI init [ 0.668265] pnp: PnP ACPI: found 6 devices [ 0.689969] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.693209] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.695249] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.697333] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.699671] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.701899] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.704866] NET: Registered protocol family 2 [ 0.707370] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.712327] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.715949] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.721290] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.724520] TCP: Hash tables configured (established 65536 bind 65536) [ 0.727492] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.730289] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.733115] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.736224] NET: Registered protocol family 1 [ 0.738702] RPC: Registered named UNIX socket transport module. [ 0.740887] RPC: Registered udp transport module. [ 0.742677] RPC: Registered tcp transport module. [ 0.744374] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.746836] NET: Registered protocol family 44 [ 0.748035] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.749474] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.751063] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.752546] PCI: CLS 0 bytes, default 64 [ 0.753727] Unpacking initramfs... [ 2.189741] debug: unmapping init [mem 0xffff99a6bcc64000-0xffff99a6bffcffff] [ 2.193529] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.195414] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.198260] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.715121] Initialise system trusted keyrings [ 2.716955] Key type blacklist registered [ 2.719072] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.728220] zbud: loaded [ 2.731418] *** VALIDATE nfs *** [ 2.732607] *** VALIDATE nfs4 *** [ 2.734161] pstore: using deflate compression [ 2.737501] Platform Keyring initialized [ 2.844070] NET: Registered protocol family 38 [ 2.845251] Key type asymmetric registered [ 2.846510] Asymmetric key parser 'x509' registered [ 2.847756] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.849793] io scheduler mq-deadline registered [ 2.850697] io scheduler kyber registered [ 2.851605] io scheduler bfq registered [ 2.853256] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.855147] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.857126] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.859525] ACPI: Power Button [PWRF] [ 2.864177] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.869369] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.878640] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.907866] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.936204] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.940541] Non-volatile memory driver v1.3 [ 2.942365] Linux agpgart interface v0.103 [ 2.974303] virtio_blk virtio1: [vda] 145904 512-byte logical blocks (74.7 MB/71.2 MiB) [ 2.977386] vda: detected capacity change from 0 to 74702848 [ 2.994127] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 2.997155] vdb: detected capacity change from 0 to 1073741824 [ 3.003172] libphy: Fixed MDIO Bus: probed [ 3.018095] usbcore: registered new interface driver usbserial_generic [ 3.019431] usbserial: USB Serial support registered for generic [ 3.020783] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.023354] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.024461] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.026399] mousedev: PS/2 mouse device common for all mice [ 3.029048] rtc_cmos 00:05: RTC can wake from S4 [ 3.031522] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.033616] rtc_cmos 00:05: registered as rtc0 [ 3.036891] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.039385] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.040705] intel_pstate: CPU model not supported [ 3.044975] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.049160] hid: raw HID events driver (C) Jiri Kosina [ 3.051084] usbcore: registered new interface driver usbhid [ 3.053065] usbhid: USB HID core driver [ 3.054704] drop_monitor: Initializing network drop monitor service [ 3.057278] Initializing XFRM netlink socket [ 3.059295] NET: Registered protocol family 10 [ 3.062028] Segment Routing with IPv6 [ 3.062973] NET: Registered protocol family 17 [ 3.064420] mpls_gso: MPLS GSO support [ 3.069498] RAS: Correctable Errors collector initialized. [ 3.070814] AVX version of gcm_enc/dec engaged. [ 3.071783] AES CTR mode by8 optimization enabled [ 3.135170] sched_clock: Marking stable (3135152371, 0)->(4026623942, -891471571) [ 3.137312] registered taskstats version 1 [ 3.139382] Loading compiled-in X.509 certificates [ 3.140506] zswap: loaded using pool lzo/zbud [ 3.155798] Key type big_key registered [ 3.164828] Key type encrypted registered [ 3.165923] ima: No TPM chip found, activating TPM-bypass! [ 3.167088] ima: Allocated hash algorithm: sha1 [ 3.168311] ima: No architecture policies found [ 3.169389] evm: Initialising EVM extended attributes: [ 3.170502] evm: security.selinux [ 3.171375] evm: security.ima [ 3.172386] evm: security.capability [ 3.173384] evm: HMAC attrs: 0x1 [ 3.175198] rtc_cmos 00:05: setting system clock to 2026-08-15 21:43:16 UTC (1786830196) [ 3.179230] debug: unmapping init [mem 0xffffffffaa003000-0xffffffffaa1fffff] [ 3.181342] debug: unmapping init [mem 0xffffffffa8d82000-0xffffffffa9058fff] [ 3.192251] Write protecting the kernel read-only data: 28672k [ 3.194971] debug: unmapping init [mem 0xffffffffa7403000-0xffffffffa75fffff] [ 3.197101] debug: unmapping init [mem 0xffffffffa7d14000-0xffffffffa7dfffff] [ 3.226923] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.235537] systemd[1]: Detected virtualization kvm. [ 3.237436] systemd[1]: Detected architecture x86-64. [ 3.239910] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.265268] systemd[1]: No hostname configured. [ 3.266776] systemd[1]: Set hostname to . [ 3.268605] random: systemd: uninitialized urandom read (16 bytes read) [ 3.271078] systemd[1]: Initializing machine ID from random generator. [ 3.314180] random: ln: uninitialized urandom read (6 bytes read) [ 3.397551] random: systemd: uninitialized urandom read (16 bytes read) [ 3.400270] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 3.403837] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 3.409885] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ OK ] Reached target Timers. [ OK ] Reached target Swap. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Control Socket. Starting Setup Virtual Console... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Apply Kernel Variables... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Initrd Root Device. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 3.992288] device-mapper: uevent: version 1.0.3 [ 3.994111] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 4.615096] virtio_net virtio0 ens2: renamed from eth0 [ 4.676184] scsi host0: ata_piix [ 4.799299] scsi host1: ata_piix [ 4.800624] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 4.803026] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 8.317448] dracut-initqueue[580]: RTNETLINK answers: File exists [ 9.678780] random: crng init done [ 9.680189] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 9.937691] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Apply Kernel Variables. Stopping udev Kernel Device Manager... [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.025884] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.296748] SELinux: Disabled at runtime. [ 11.355185] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.362862] systemd[1]: Detected virtualization kvm. [ 11.364402] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 11.810827] systemd[1]: initrd-switch-root.service: Succeeded. [ 11.813846] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 11.818463] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 11.822432] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 11.825722] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 11.832357] systemd[1]: Starting Journal Service... Starting Journal Service... [ 11.838220] systemd[1]: Created slice system-getty.slice. [ OK ] Created slice system-getty.slice. Mounting Huge Pages File System... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Starting Apply Kernel Variables... Mounting POSIX Message Queue File System... [ OK ] Listening on udev Control Socket. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on udev Kernel Socket. Activating swap /dev/disk/by-label/SWAP... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 11.916079] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on Process Core Dump Socket. Starting Create list of required st…ce nodes for the current kernel... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Starting udev Coldplug all Devices... [ OK ] Stopped target Switch Root. Starting Remount Root and Kernel File Systems... [ OK ] Stopped target Initrd Root File System. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. Mounting Kernel Debug File System... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Reached target rpc_pipefs.target. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Stopped target Initrd File Systems. [ OK ] Started Journal Service. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Kernel Debug File System. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 12.218533] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 12.536666] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 12.576328] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 12.624793] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 12.640794] EDAC sbridge: Ver: 1.1.2 [ 13.658413] Key type dns_resolver registered [ 13.957650] NFS: Registering the id_resolver key type [ 13.959629] Key type id_resolver registered [ 13.961118] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Mark the need to relabel after reboot... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Basic System. Starting Login Service... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg247-client login: [ 71.216162] libcfs: loading out-of-tree module taints kernel. [ 71.285097] Key type ._llcrypt registered [ 71.312874] Key type .llcrypt registered [ 72.377959] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 72.407194] alg: No test for adler32 (adler32-zlib) [ 73.885450] Lustre: Lustre: Build Version: 2.17.57_3_g67c4c6e [ 74.825119] LNet: Added LNI 192.168.202.47@tcp [8/256/0/180] [ 75.243101] hrtimer: interrupt took 2176775 ns [ 76.792281] Key type lgssc registered [ 78.618797] Lustre: Echo OBD driver; http://www.lustre.org/ [ 232.749967] Lustre: Mounted lustre-client - version 2.17.57_3_g67c4c6e [ 238.775308] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 253.514793] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing check_logdir /tmp/testlogs/ [ 258.530385] Lustre: lustre-OST0000-osc-ffff99a7031ab000: disconnect after 23s idle [ 260.452525] Lustre: DEBUG MARKER: oleg247-client.virtnet: executing yml_node [ 265.307634] Lustre: DEBUG MARKER: Client: 2.17.57.3 [ 268.159692] Lustre: DEBUG MARKER: MDS: 2.17.57.3 [ 271.103641] Lustre: DEBUG MARKER: OSS: 2.17.57.3 [ 273.188196] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Sat Aug 15 17:47:44 EDT 2026 [ 294.696515] Lustre: Unmounted lustre-client [ 327.809276] Key type lgssc unregistered [ 328.133040] LNet: 8418:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 328.136879] LNetError: 8418:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 328.158907] LNet: Removed LNI 192.168.202.47@tcp [ 329.409200] Key type .llcrypt unregistered [ 329.413063] Key type ._llcrypt unregistered [ 361.216295] Key type ._llcrypt registered [ 361.220206] Key type .llcrypt registered [ 361.379046] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 361.390402] alg: No test for adler32 (adler32-zlib) [ 362.248638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 362.342685] LNet: Added LNI 192.168.202.47@tcp [8/256/0/180] [ 362.367884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 366.974182] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 367.658751] LNet: 9310:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 367.664799] LNetError: 9310:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 367.685403] LNet: Removed LNI 192.168.202.47@tcp [ 368.995483] Key type .llcrypt unregistered [ 368.997320] Key type ._llcrypt unregistered [ 377.683769] Key type ._llcrypt registered [ 377.689073] Key type .llcrypt registered [ 379.349481] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 17:49:30 (1786830570) [ 379.632649] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 379.651905] alg: No test for adler32 (adler32-zlib) [ 380.791481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 380.989512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 381.042786] LNet: 9740:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 381.085715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 381.218688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 381.260873] LNet: 9747:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 391.035104] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 17:49:42 (1786830582) [ 391.181560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 391.218878] LNet: 10321:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 391.252139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 391.387933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 398.747474] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 17:49:50 (1786830590) [ 398.934962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 398.984840] LNet: 10908:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 399.041395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 399.162295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 406.713514] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 17:49:58 (1786830598) [ 406.851969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 406.935858] LNet: 11493:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 406.963379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 407.061115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 414.519905] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 17:50:06 (1786830606) [ 414.637326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 414.702124] LNet: 12077:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 414.743199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 414.879435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 415.462082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 422.449094] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 17:50:14 (1786830614) [ 422.566364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 422.657674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 422.752828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 423.307941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 423.872966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 431.915690] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 17:50:23 (1786830623) [ 432.095793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 432.119466] LNet: 13265:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 432.128093] LNet: 13265:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 432.172248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 432.282354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 439.799861] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 17:50:31 (1786830631) [ 439.897987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 439.955785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 440.061286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 440.128279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 440.592954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 440.643245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 441.095511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 441.197032] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 441.766494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 441.790892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 442.411943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 442.469869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 442.970915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 443.005862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 443.377810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 443.399562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 443.868891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 443.908721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 444.611385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 444.669362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 445.036242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 445.108882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 445.683080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 445.737403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 446.117786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 446.146950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 446.556523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 446.598817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 454.001951] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 17:50:45 (1786830645) [ 454.293312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 454.310519] LNet: 14536:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 454.314457] LNet: 14536:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 454.332847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 454.387791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 454.410592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 462.150260] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 17:50:53 (1786830653) [ 462.333529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 462.426362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 462.553190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 462.582427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 471.648266] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 17:51:03 (1786830663) [ 471.994208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 472.057737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 472.153317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 472.215941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 480.502393] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 17:51:12 (1786830672) [ 480.609394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 480.676110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 480.847709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 480.896095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 490.775431] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 17:51:21 (1786830681) [ 490.858038] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 490.909367] LNet: 16875:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 490.922840] LNet: 16875:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 490.980891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 491.154207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 491.225450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 500.894863] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 17:51:32 (1786830692) [ 501.261812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 501.308808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 501.384630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 501.421614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 511.187055] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 17:51:42 (1786830702) [ 511.459765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 511.586186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 511.800372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 512.349280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 521.081680] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 17:51:52 (1786830712) [ 521.232221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 521.296424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 521.557375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 521.872382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 529.780909] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 17:52:01 (1786830721) [ 529.899206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 529.976885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 530.121385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 530.492818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 538.943260] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 17:52:10 (1786830730) [ 539.187050] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 539.298148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 540.040777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 540.594658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 549.704943] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 17:52:21 (1786830741) [ 549.886986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 549.983227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 550.140231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 550.221217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 558.026206] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 17:52:29 (1786830749) [ 558.309530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 558.360483] LNet: 21132:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 558.372958] LNet: 21132:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 6 previous similar messages [ 558.405971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 558.517809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 558.559773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 567.011382] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 17:52:38 (1786830758) [ 567.157743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 567.252825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 567.429555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 575.326947] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 17:52:47 (1786830767) [ 575.581947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 575.638148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 575.755690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 582.939379] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 17:52:55 (1786830775) [ 583.064164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 583.142975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 583.250656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 583.956490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 592.139182] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 17:53:03 (1786830783) [ 592.292769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 592.410847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 592.535776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 592.555719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 599.333995] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 17:53:11 (1786830791) [ 599.652511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 599.737382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 599.980919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 600.080714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 607.531827] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 17:53:19 (1786830799) [ 607.714734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 607.782738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 607.855962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 607.919383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 615.971947] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 17:53:27 (1786830807) [ 616.388552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 616.523710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 616.679527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 616.793592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 626.295315] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 17:53:38 (1786830818) [ 626.650341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 626.797852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 626.914295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 627.032123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 627.066942] LNetError: 25861:0:(peer.c:2362:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 627.172954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 627.254173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 634.107278] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 17:53:45 (1786830825) [ 634.331188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 634.370856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 634.471616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.7Dk8mJ6leZ [ 634.608476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.7Dk8mJ6leZ [ 634.665494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.7Dk8mJ6leZ [ 641.981375] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 17:53:53 (1786830833) [ 642.105890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 642.157640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 642.224906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 642.295985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 642.352513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 650.868841] Lustre: DEBUG MARKER: == sanity-lnet test 40: LNetAddPeer merges a pre-existing non-primary peer NID ========================================================== 17:54:02 (1786830842) [ 650.957309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 651.045599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 651.135795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 651.167843] LNet: Added LNI 192.168.202.47@tcp [8/256/0/180] [ 653.221569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.47@tcp [ 655.233081] Lustre: Lustre: Build Version: 2.17.57_3_g67c4c6e [ 657.615719] Key type lgssc registered [ 659.395563] Lustre: Echo OBD driver; http://www.lustre.org/ [ 663.840212] LNetError: 27623:0:(lib-move.c:2158:lnet_handle_find_routed_path()) peer 192.168.202.47@tcp99 has no available nets [ 666.882474] Key type lgssc unregistered [ 667.047891] LNetError: 28733:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 667.058335] LNet: Removed LNI 192.168.202.47@tcp [ 667.851886] Key type .llcrypt unregistered [ 667.856076] Key type ._llcrypt unregistered [ 676.089277] Key type ._llcrypt registered [ 676.090945] Key type .llcrypt registered [ 677.518696] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 17:54:29 (1786830869) [ 677.841662] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 677.871044] alg: No test for adler32 (adler32-zlib) [ 678.817688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 678.878874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 681.454285] LNet: Message forwarding will be enabled in 2s [ 684.511948] LNet: Message forwarding enabled [ 685.326671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 685.352286] LNet: Message forwarding will stop in 2s to 6s [ 688.815720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 690.444911] LNet: Message forwarding disabled [ 690.638594] LNet: Message forwarding will be enabled in 2s [ 694.562037] LNet: Message forwarding enabled [ 698.472651] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 17:54:50 (1786830890) [ 698.777685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 698.825645] LNet: 30132:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 700.220472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 700.327231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 706.684995] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 17:54:58 (1786830898) [ 706.971042] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 707.007830] LNet: 30714:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 707.036637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 707.093751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 707.139974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 707.179798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 707.219696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 707.251562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 707.282770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 707.319158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 707.358405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 707.390474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 707.425853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 707.452836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 707.499965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 707.540591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 707.584314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 707.646973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 707.693777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 707.724294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 707.756470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 707.786698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 707.809470] LNetError: 30757:0:(router.c:625:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 714.509721] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 17:55:06 (1786830906) [ 714.927157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 714.977582] LNet: 31329:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 715.007510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 715.072705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 722.659427] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 17:55:14 (1786830914) [ 726.058604] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg247-server [ 728.311690] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 17:55:19 (1786830919) [ 728.562211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 728.609153] LNet: 32260:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 728.639405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 728.682341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 728.708526] LNet: Added LNI 192.168.202.47@tcp [8/256/0/180] [ 730.009859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.202.[48-52]@tcp [ 730.018948] LNet: 32393:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 730.073181] LNetError: 32264:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.202.51@tcp is being used as a gateway but routing feature is not turned on [ 730.095197] LNetError: 32264:0:(lib-lnet.h:1279:lnet_set_route_aliveness()) route to tcp1 through 192.168.202.51@tcp has gone from up to down [ 731.045076] LNetError: 32264:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.202.48@tcp is being used as a gateway but routing feature is not turned on [ 731.056551] LNetError: 32264:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 732.078422] LNetError: 32264:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.202.48@tcp is being used as a gateway but routing feature is not turned on [ 732.095958] LNetError: 32264:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 4 previous similar messages [ 734.120909] LNetError: 32264:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.202.51@tcp is being used as a gateway but routing feature is not turned on [ 734.155781] LNetError: 32264:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 738.273982] LNetError: 32264:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.202.48@tcp is being used as a gateway but routing feature is not turned on [ 738.290526] LNetError: 32264:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 19 previous similar messages [ 739.494739] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 17:55:30 (1786830930) [ 741.438499] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg247-server [ 743.017433] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 17:55:35 (1786830935) [ 743.247403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 743.305467] LNet: 33327:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 743.326225] LNetError: 33327:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 743.358445] LNet: Removed LNI 192.168.202.47@tcp [ 743.428563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 743.512437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 743.543554] LNet: Added LNI 192.168.202.47@tcp [8/256/0/180] [ 744.684126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.202.[48-53/2]@tcp [ 744.715628] LNet: 33448:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 744.726646] LNet: 33448:0:(router.c:729:lnet_add_route()) Skipped 4 previous similar messages [ 744.760754] LNetError: 33335:0:(lib-lnet.h:1279:lnet_set_route_aliveness()) route to tcp103 through 192.168.202.50@tcp has gone from up to down [ 744.784621] LNetError: 33335:0:(lib-lnet.h:1279:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 744.803205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.202.[48-53/2]@tcp [ 752.208322] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 17:55:43 (1786830943) [ 752.349379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 752.386961] LNet: 34039:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 752.402190] LNetError: 34039:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 752.433103] LNet: Removed LNI 192.168.202.47@tcp [ 752.457560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 752.580254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 752.624548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 752.649965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 752.696594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 752.727623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 752.750799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 752.788773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 752.872865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 752.916285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 752.996913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 753.085228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 753.117551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 753.164187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 753.252069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 753.293512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 753.366761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 753.438175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 753.474348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 753.535769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 753.624254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 753.659603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 753.724517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 753.816223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 753.861552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 753.912073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 753.999403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 754.039504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 754.102369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 754.199856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 754.245166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 754.308146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 754.365578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 754.407225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 754.488180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 761.476121] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 17:55:53 (1786830953) [ 763.629135] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg247-server [ 765.582717] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 17:55:57 (1786830957) [ 768.213977] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg247-server [ 770.624212] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 17:56:02 (1786830962) [ 770.759593] LNet: 35434:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 770.767293] LNet: 35434:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 11 previous similar messages [ 771.916813] Key type .llcrypt unregistered [ 771.926965] Key type ._llcrypt unregistered [ 780.065800] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 780.267828] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 780.275987] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 780.286033] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 783.043068] Key type ._llcrypt registered [ 783.045624] Key type .llcrypt registered [ 783.363634] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 783.406180] alg: No test for adler32 (adler32-zlib) [ 784.351497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 784.417496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 784.450386] LNet: Added LNI 192.168.202.47@tcp [8/256/0/180] [ 785.649885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 785.693243] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 788.230587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 788.256241] LNet: Removed LNI 10.1.2.3@tcp [ 788.474325] LNet: 36164:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 788.480384] LNetError: 36164:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 788.497907] LNet: Removed LNI 192.168.202.47@tcp [ 789.365716] Key type .llcrypt unregistered [ 789.367397] Key type ._llcrypt unregistered [ 804.289530] Key type ._llcrypt registered [ 804.295956] Key type .llcrypt registered [ 806.339202] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 17:56:37 (1786830997) [ 807.035369] Key type .llcrypt unregistered [ 807.045934] Key type ._llcrypt unregistered [ 815.978660] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 816.273481] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 816.287279] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 816.297766] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 817.894493] Key type ._llcrypt registered [ 817.904781] Key type .llcrypt registered [ 818.101921] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 818.121711] alg: No test for adler32 (adler32-zlib) [ 819.041872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 819.122596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 819.179682] LNet: Added LNI 192.168.202.47@tcp [8/256/0/180] [ 820.016364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 820.044805] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 823.435692] LNet: 37654:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 823.441384] LNetError: 37654:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 823.456259] LNet: Removed LNI 10.1.2.3@tcp [ 823.470517] LNet: Removed LNI 192.168.202.47@tcp [ 824.259054] Key type .llcrypt unregistered [ 824.263090] Key type ._llcrypt unregistered [ 839.644591] Key type ._llcrypt registered [ 839.646152] Key type .llcrypt registered [ 841.085291] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 17:57:12 (1786831032) [ 841.853140] Key type .llcrypt unregistered [ 841.865688] Key type ._llcrypt unregistered [ 850.726952] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 850.963179] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 850.971213] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 850.979299] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 853.757518] Key type ._llcrypt registered [ 853.760020] Key type .llcrypt registered [ 853.991628] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 854.023602] alg: No test for adler32 (adler32-zlib) [ 855.084326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 855.213300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 855.299069] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 858.042261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 858.072793] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 861.187167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 861.216103] LNet: Removed LNI 10.1.2.3@tcp [ 861.247762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 861.277816] LNet: Removed LNI 10.1.2.31@tcp [ 861.297793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 861.335494] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 862.332557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 862.350570] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 863.360934] LNet: 39589:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 863.366686] LNetError: 39589:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg:0 not found [ 863.374442] LNet: Removed LNI 10.1.2.31@tcp [ 863.385573] LNet: Removed LNI 10.1.2.3@tcp [ 864.118418] Key type .llcrypt unregistered [ 864.121317] Key type ._llcrypt unregistered [ 878.153392] Key type ._llcrypt registered [ 878.155694] Key type .llcrypt registered [ 879.638984] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 17:57:51 (1786831071) [ 880.157502] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 880.184528] alg: No test for adler32 (adler32-zlib) [ 881.234841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 881.381321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 881.439069] LNet: Added LNI 192.168.202.47@tcp [8/256/0/180] [ 882.563239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 882.579968] LNet: 40547:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 882.584981] LNetError: 40547:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 882.594394] LNet: Removed LNI 192.168.202.47@tcp [ 882.613906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 882.703836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.202.47@tcp [ 882.727852] LNet: Added LNI 192.168.202.47@tcp [8/256/0/180] [ 883.568169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.202.47@tcp [ 883.632980] LNet: Removed LNI 192.168.202.47@tcp [ 883.732575] LNet: 40682:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 884.452584] Key type .llcrypt unregistered [ 884.455926] Key type ._llcrypt unregistered [ 897.531134] Key type ._llcrypt registered [ 897.532609] Key type .llcrypt registered [ 898.870134] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 17:58:10 (1786831090) [ 901.420738] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg247-server [ 903.934545] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 17:58:15 (1786831095) [ 904.724567] Key type .llcrypt unregistered [ 904.728913] Key type ._llcrypt unregistered [ 912.773210] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 913.032423] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 913.039363] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 913.050778] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 915.788602] Key type ._llcrypt registered [ 915.790355] Key type .llcrypt registered [ 916.077236] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 916.114757] alg: No test for adler32 (adler32-zlib) [ 917.167774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 917.272088] LNet: Added LNI 192.168.202.47@tcp1 [8/256/0/180] [ 918.133787] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 920.889796] LNet: 42524:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 920.896103] LNetError: 42524:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 920.906322] LNet: Removed LNI 192.168.202.47@tcp1 [ 920.920644] LNet: Removed LNI 10.1.2.3@tcp2 [ 921.656721] Key type .llcrypt unregistered [ 921.659151] Key type ._llcrypt unregistered [ 935.991241] Key type ._llcrypt registered [ 935.994181] Key type .llcrypt registered [ 938.144609] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 17:58:49 (1786831129) [ 938.555276] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 938.576853] alg: No test for adler32 (adler32-zlib) [ 939.593341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 939.695194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 939.746197] LNet: Added LNI 192.168.202.47@tcp [8/256/0/180] [ 941.084914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 941.102829] LNet: 43499:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 941.119410] LNetError: 43499:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 941.141615] LNet: Removed LNI 192.168.202.47@tcp [ 941.160773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 941.254679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 941.300897] LNet: Added LNI 192.168.202.47@tcp [8/256/0/180] [ 949.534606] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 17:59:01 (1786831141) [ 949.845788] LNet: 44091:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 949.853573] LNetError: 44091:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 949.861996] LNet: Removed LNI 192.168.202.47@tcp [ 950.809536] Key type .llcrypt unregistered [ 950.812481] Key type ._llcrypt unregistered [ 958.288857] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 958.457354] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 958.474737] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 958.485798] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 961.254624] Key type ._llcrypt registered [ 961.256916] Key type .llcrypt registered [ 961.429618] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 961.457722] alg: No test for adler32 (adler32-zlib) [ 962.409416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 962.458572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 962.482312] LNet: Added LNI 192.168.202.47@tcp [8/256/0/180] [ 963.269558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 963.314211] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 965.625081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 965.647912] LNet: 44832:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 965.653421] LNetError: 44832:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 965.657417] LNet: Removed LNI 10.1.2.3@tcp [ 965.666781] LNet: Removed LNI 192.168.202.47@tcp [ 965.687104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 965.775662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 965.801541] LNet: Added LNI 192.168.202.47@tcp [8/256/0/180] [ 965.806360] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 966.622290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 966.645548] LNet: 44859:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 966.653727] LNetError: 44859:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 966.659568] LNetError: 44859:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 966.663846] LNet: Removed LNI 10.1.2.3@tcp [ 966.672247] LNet: Removed LNI 192.168.202.47@tcp [ 966.706414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 966.785578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 966.817219] LNet: Added LNI 192.168.202.47@tcp [8/256/0/180] [ 966.841851] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 967.560727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 967.602388] LNet: Added LNI 192.168.202.47@tcp2 [8/256/0/180] [ 968.957569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 968.973542] LNet: 45022:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 968.978473] LNetError: 45022:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 968.981449] LNetError: 45022:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 969.004347] LNet: Removed LNI 10.1.2.3@tcp [ 969.006662] LNet: Removed LNI 192.168.202.47@tcp [ 969.024394] LNet: Removed LNI 192.168.202.47@tcp2 [ 969.073919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 969.134168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 969.161413] LNet: Added LNI 192.168.202.47@tcp [8/256/0/180] [ 969.167063] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 969.173854] LNet: Added LNI 192.168.202.47@tcp2 [8/256/0/180] [ 978.445619] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 17:59:29 (1786831169) [ 978.639196] LNet: 45612:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 978.647629] LNetError: 45612:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 978.659800] LNetError: 45612:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 978.678247] LNet: Removed LNI 10.1.2.3@tcp [ 978.682174] LNet: Removed LNI 192.168.202.47@tcp [ 978.701173] LNet: Removed LNI 192.168.202.47@tcp2 [ 979.649148] Key type .llcrypt unregistered [ 979.654486] Key type ._llcrypt unregistered [ 988.290637] Key type ._llcrypt registered [ 988.292337] Key type .llcrypt registered [ 988.774686] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 988.829864] alg: No test for adler32 (adler32-zlib) [ 989.825720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 989.893752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 989.929558] LNet: Added LNI 192.168.202.47@tcp [8/256/0/180] [ 990.693257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 990.707892] LNet: 46018:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 990.713732] LNetError: 46018:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 990.728110] LNet: Removed LNI 192.168.202.47@tcp [ 990.764811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 990.856550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 990.888988] LNet: Added LNI 192.168.202.47@tcp [8/256/0/180] [ 991.548626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 991.580186] LNet: 46048:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 991.589559] LNetError: 46048:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 991.640313] LNet: Removed LNI 192.168.202.47@tcp [ 991.700660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 991.780985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 991.817627] LNet: Added LNI 192.168.202.47@tcp [8/256/0/180] [ 992.479260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 992.519786] LNet: Removed LNI 192.168.202.47@tcp [ 992.553681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 992.623781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 992.653201] LNet: Added LNI 192.168.202.47@tcp [8/256/0/180] [ 993.381114] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 993.522449] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 993.533774] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 993.548900] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 995.153327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 995.189921] LNet: 46151:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 995.194281] LNet: 46151:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 995.214064] LNetError: 46151:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 995.222291] LNetError: 46151:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 995.255956] LNet: Removed LNI 192.168.202.47@tcp [ 995.308403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 995.373033] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 995.448113] LNet: Added LNI 192.168.202.47@tcp [8/256/0/180] [ 1003.318123] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 17:59:55 (1786831195) [ 1003.557252] LNet: 46748:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1003.563387] LNetError: 46748:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1003.577274] LNet: Removed LNI 192.168.202.47@tcp [ 1004.366863] Key type .llcrypt unregistered [ 1004.370310] Key type ._llcrypt unregistered [ 1011.870770] Key type ._llcrypt registered [ 1011.874729] Key type .llcrypt registered [ 1012.236725] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1012.257350] alg: No test for adler32 (adler32-zlib) [ 1013.229708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1013.347395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1013.402077] LNet: Added LNI 192.168.202.47@tcp [8/256/0/180] [ 1014.632137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1014.652863] LNet: 47151:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1014.657584] LNetError: 47151:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1014.678332] LNet: Removed LNI 192.168.202.47@tcp [ 1014.711438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1014.832478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 1014.861721] LNet: Added LNI 192.168.202.47@tcp [8/256/0/180] [ 1023.245370] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 18:00:15 (1786831215) [ 1023.492362] LNet: 47743:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1023.500151] LNetError: 47743:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1023.511279] LNet: Removed LNI 192.168.202.47@tcp [ 1024.258562] Key type .llcrypt unregistered [ 1024.260479] Key type ._llcrypt unregistered [ 1031.689656] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1031.873411] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1031.876796] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1031.880462] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1034.509235] Key type ._llcrypt registered [ 1034.511730] Key type .llcrypt registered [ 1034.730568] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1034.747825] alg: No test for adler32 (adler32-zlib) [ 1035.755613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1035.816430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1035.844600] LNet: Added LNI 192.168.202.47@tcp [8/256/0/180] [ 1036.685376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1036.709697] LNet: 48201:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1036.715894] LNetError: 48201:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1036.729730] LNet: Removed LNI 192.168.202.47@tcp [ 1036.746820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1036.837495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 1036.862886] LNet: Added LNI 192.168.202.47@tcp [8/256/0/180] [ 1043.716883] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 18:00:35 (1786831235) [ 1043.933694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1043.947891] LNet: 48799:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1043.952805] LNetError: 48799:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1043.964427] LNet: Removed LNI 192.168.202.47@tcp [ 1043.984328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1044.097328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 1050.764821] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 18:00:42 (1786831242) [ 1050.892600] LNet: 49376:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1051.657322] Key type .llcrypt unregistered [ 1051.663547] Key type ._llcrypt unregistered [ 1059.269210] Key type ._llcrypt registered [ 1059.273573] Key type .llcrypt registered [ 1059.682193] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1059.714738] alg: No test for adler32 (adler32-zlib) [ 1060.802348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1060.889386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1060.950830] LNet: Added LNI 192.168.202.47@tcp [8/256/0/180] [ 1062.054504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1062.077688] LNet: 49784:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1062.083657] LNetError: 49784:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1062.099753] LNet: Removed LNI 192.168.202.47@tcp [ 1062.134366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1062.264290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 1062.295670] LNet: Added LNI 192.168.202.47@tcp [8/256/0/180] [ 1070.756130] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 18:01:02 (1786831262) [ 1071.073597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1071.116157] LNet: 50392:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1071.126297] LNetError: 50392:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1071.142916] LNet: Removed LNI 192.168.202.47@tcp [ 1071.189937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1071.289940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 1078.155747] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 18:01:09 (1786831269) [ 1080.102932] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 1081.951104] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 18:01:13 (1786831273) [ 1083.515253] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 1085.423077] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 18:01:17 (1786831277) [ 1087.351817] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 1089.703511] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 18:01:21 (1786831281) [ 1091.525890] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 1094.062740] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 18:01:25 (1786831285) [ 1095.901639] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 1098.125890] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing ========================================================== 18:01:29 (1786831289) [ 1098.498850] LNet: 52447:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1099.611804] Key type .llcrypt unregistered [ 1099.615541] Key type ._llcrypt unregistered [ 1107.617473] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1107.790429] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1107.800556] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1107.803317] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1110.424740] Key type ._llcrypt registered [ 1110.427166] Key type .llcrypt registered [ 1110.597335] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1110.616674] alg: No test for adler32 (adler32-zlib) [ 1111.578544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1111.664992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1111.690317] LNet: Added LNI 192.168.202.47@tcp [8/512/0/180] [ 1111.701165] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 1111.704456] LNet: 52773:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1111.831540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1111.864509] LNet: 52792:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1111.873582] LNetError: 52792:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1111.880517] LNet: Removed LNI 10.1.2.3@tcp [ 1111.883196] LNetError: 52792:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1111.895636] LNet: Removed LNI 192.168.202.47@tcp [ 1111.932666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1111.998149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1112.033207] LNet: Added LNI 192.168.202.47@tcp [8/256/0/180] [ 1112.133161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1112.171521] LNet: 52816:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1112.180272] LNetError: 52816:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1112.201657] LNet: Removed LNI 192.168.202.47@tcp [ 1112.226429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1112.315596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1112.339125] LNet: Added LNI 192.168.202.47@tcp [8/256/0/180] [ 1112.348503] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1112.353185] LNet: 52824:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1112.495421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1112.525604] LNet: 52842:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1112.535110] LNetError: 52842:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1112.552677] LNet: Removed LNI 10.1.2.3@tcp [ 1112.554112] LNetError: 52842:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1112.577636] LNet: Removed LNI 192.168.202.47@tcp [ 1112.607694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1112.708543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1112.743100] LNet: Added LNI 192.168.202.47@tcp [8/256/0/180] [ 1112.750204] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1112.751959] LNet: 52850:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1112.985700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1113.018769] LNet: 52876:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1113.022182] LNetError: 52876:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1113.027921] LNet: Removed LNI 10.1.2.3@tcp [ 1113.030155] LNetError: 52876:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1113.070525] LNet: Removed LNI 192.168.202.47@tcp [ 1113.087808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1113.174672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1113.224274] LNet: Added LNI 192.168.202.47@tcp [8/256/0/182] [ 1113.230510] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 1113.558680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1113.622350] LNet: 52904:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1113.631325] LNetError: 52904:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1113.635628] LNet: Removed LNI 10.1.2.3@tcp [ 1113.637868] LNetError: 52904:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1113.652913] LNet: Removed LNI 192.168.202.47@tcp [ 1113.671857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1113.763168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1113.816959] LNet: Added LNI 192.168.202.47@tcp [8/256/0/182] [ 1113.822865] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 1113.830688] LNet: 52912:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1113.913668] LNet: Removed LNI 10.1.2.3@tcp [ 1113.927869] LNet: Removed LNI 192.168.202.47@tcp [ 1114.658471] Key type .llcrypt unregistered [ 1114.663846] Key type ._llcrypt unregistered [ 1129.144809] Key type ._llcrypt registered [ 1129.147723] Key type .llcrypt registered [ 1130.729147] Lustre: DEBUG MARKER: == sanity-lnet test 180: FastReg map error should not leak descriptors ========================================================== 18:02:02 (1786831322) [ 1132.814465] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_180 Need o2ib NETTYPE [ 1134.889653] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 18:02:06 (1786831326) [ 1135.520867] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1135.547454] alg: No test for adler32 (adler32-zlib) [ 1136.543371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1136.657638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1136.704570] LNet: Added LNI 192.168.202.47@tcp [8/256/0/180] [ 1137.693041] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.202.47 [ 1137.728851] LNet: 54181:0:(config.c:1738:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1137.745124] LNet: Removed LNI 192.168.202.47@tcp [ 1145.543859] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 18:02:17 (1786831337) [ 1145.786203] LNet: 54751:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1145.796414] LNetError: 54751:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1145.814598] LNet: Removed LNI 192.168.202.47@tcp [ 1146.616070] Key type .llcrypt unregistered [ 1146.617756] Key type ._llcrypt unregistered [ 1153.375788] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1153.550520] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1156.442868] Key type ._llcrypt registered [ 1156.446479] Key type .llcrypt registered [ 1156.754944] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1156.777312] alg: No test for adler32 (adler32-zlib) [ 1158.345555] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1165.980917] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 18:02:37 (1786831357) [ 1166.260485] LNet: 55669:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1166.271058] LNetError: 55669:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1166.300663] LNet: Removed LNI 10.1.2.3@tcp [ 1167.180758] Key type .llcrypt unregistered [ 1167.182569] Key type ._llcrypt unregistered [ 1174.474873] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1174.706339] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1174.719960] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1174.727414] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1177.565860] Key type ._llcrypt registered [ 1177.568884] Key type .llcrypt registered [ 1177.910166] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1177.924859] alg: No test for adler32 (adler32-zlib) [ 1178.943163] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1184.897382] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 18:02:56 (1786831376) [ 1186.433300] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 1188.252804] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 18:03:00 (1786831380) [ 1188.420442] LNet: 56867:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1188.429781] LNetError: 56867:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1188.456129] LNet: Removed LNI 10.1.2.3@tcp [ 1189.353512] Key type .llcrypt unregistered [ 1189.357648] Key type ._llcrypt unregistered [ 1196.090356] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1196.386236] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1196.400307] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1196.411307] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1199.454233] Key type ._llcrypt registered [ 1199.461611] Key type .llcrypt registered [ 1199.613182] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1199.625331] alg: No test for adler32 (adler32-zlib) [ 1200.537552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1200.599352] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1208.545661] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 18:03:20 (1786831400) [ 1209.001922] LNet: 58103:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1209.018893] LNetError: 58103:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1209.031255] LNet: Removed LNI 10.1.2.3@tcp [ 1209.571769] Key type .llcrypt unregistered [ 1209.574759] Key type ._llcrypt unregistered [ 1215.389891] Key type ._llcrypt registered [ 1215.393538] Key type .llcrypt registered [ 1215.595162] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1215.613882] alg: No test for adler32 (adler32-zlib) [ 1216.532293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1216.552256] LNet: Added LNI 192.168.202.47@tcp [8/256/0/180] [ 1223.466241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.147@tcp [ 1224.122515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.47@tcp -d 192.168.202.147@tcp -m GET -e local_interrupt -r 1 [ 1224.164839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.47@tcp -d 192.168.202.47@tcp -m GET -e local_interrupt -r 1 [ 1224.230897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.147@tcp [ 1224.274669] LNet: There was an unexpected network error while writing to 192.168.202.147: rc = -22 [ 1224.499197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1224.542187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1224.588431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1224.636717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1224.687980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1224.729865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1224.735144] LNet: 1 local NIs in recovery (showing 1): 192.168.202.47@tcp [ 1224.774648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1224.809301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1224.824221] LNet: 58544:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1224.839267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1224.866288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1224.885937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1224.914565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1224.941902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1224.970262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1225.003780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1225.506543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.47@tcp -d 192.168.202.147@tcp -m GET -e local_dropped -r 1 [ 1225.541927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.47@tcp -d 192.168.202.47@tcp -m GET -e local_dropped -r 1 [ 1225.574674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.147@tcp [ 1225.592881] LNet: There was an unexpected network error while writing to 192.168.202.147: rc = -22 [ 1225.760519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1225.791660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1225.821335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1225.842941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1225.868398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1225.898374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1225.933130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1225.980923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1226.012136] LNet: 58653:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1226.033949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1226.064490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1226.101938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1226.127885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1226.157524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1226.181918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1226.214572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1226.644631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.47@tcp -d 192.168.202.147@tcp -m GET -e local_aborted -r 1 [ 1226.663260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.47@tcp -d 192.168.202.47@tcp -m GET -e local_aborted -r 1 [ 1226.687302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.147@tcp [ 1226.699369] LNet: There was an unexpected network error while writing to 192.168.202.147: rc = -22 [ 1226.703192] LNet: Skipped 1 previous similar message [ 1226.854562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1226.871065] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1226.889526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1226.907461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1226.934642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1226.957940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1226.983903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1227.006569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1227.021733] LNet: 58763:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1227.042098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1227.063467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1227.081833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1227.101715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1227.133941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1227.163460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1227.194906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1227.662728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.47@tcp -d 192.168.202.147@tcp -m GET -e local_no_route -r 1 [ 1227.679925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.47@tcp -d 192.168.202.47@tcp -m GET -e local_no_route -r 1 [ 1227.695755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.147@tcp [ 1227.927097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1227.954610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1227.980877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1228.015158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1228.042973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1228.066887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1228.088642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1228.115474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1228.142126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1228.168471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1228.186897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1228.215869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1228.241446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1228.273504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1228.303628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1228.835673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.47@tcp -d 192.168.202.147@tcp -m GET -e local_timeout -r 1 [ 1228.868955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.47@tcp -d 192.168.202.47@tcp -m GET -e local_timeout -r 1 [ 1228.901751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.147@tcp [ 1228.923767] LNet: There was an unexpected network error while writing to 192.168.202.147: rc = -22 [ 1228.932798] LNet: Skipped 2 previous similar messages [ 1229.155575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1229.185247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1229.213774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1229.246856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1229.283949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1229.310982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1229.339941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1229.366195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1229.380416] LNet: 58981:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1229.384795] LNet: 58981:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1229.402125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1229.432366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1229.462228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1229.487176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1229.514294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1229.543102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1229.562610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1230.000342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.47@tcp -d 192.168.202.147@tcp -m GET -e local_error -r 1 [ 1230.032612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.47@tcp -d 192.168.202.47@tcp -m GET -e local_error -r 1 [ 1230.067650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.147@tcp [ 1230.335964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1230.405997] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1230.454238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1230.493686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1230.548955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1230.579025] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1230.610725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1230.646535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1230.673445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1230.710801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1230.745500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1230.788536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1230.825250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1230.849926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1230.878936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1231.137691] LNet: 59135:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1231.145954] LNetError: 59135:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1232.169942] LNet: Removed LNI 192.168.202.47@tcp [ 1232.574195] Key type .llcrypt unregistered [ 1232.575735] Key type ._llcrypt unregistered [ 1248.922260] Key type ._llcrypt registered [ 1248.924646] Key type .llcrypt registered [ 1249.953948] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 18:04:02 (1786831442) [ 1250.984278] Key type .llcrypt unregistered [ 1250.986406] Key type ._llcrypt unregistered [ 1256.667613] Key type ._llcrypt registered [ 1256.674556] Key type .llcrypt registered [ 1256.879420] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1256.902642] alg: No test for adler32 (adler32-zlib) [ 1257.800517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1257.822942] LNet: Added LNI 192.168.202.47@tcp [8/256/0/180] [ 1265.367705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.147@tcp [ 1274.318990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1274.392556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1274.429481] LNet: Added LNI 192.168.202.47@tcp1 [8/256/0/180] [ 1275.714662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.47@tcp -d 192.168.202.147@tcp -m GET -e local_interrupt -r 1 [ 1275.778265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.47@tcp -d 192.168.202.147@tcp1 -m GET -e local_interrupt -r 1 [ 1275.817716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.47@tcp -d 192.168.202.47@tcp -m GET -e local_interrupt -r 1 [ 1275.846100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.47@tcp -d 192.168.202.47@tcp1 -m GET -e local_interrupt -r 1 [ 1275.887786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.47@tcp1 -d 192.168.202.147@tcp -m GET -e local_interrupt -r 1 [ 1275.938848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.47@tcp1 -d 192.168.202.147@tcp1 -m GET -e local_interrupt -r 1 [ 1275.979428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.47@tcp1 -d 192.168.202.47@tcp -m GET -e local_interrupt -r 1 [ 1276.026331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.47@tcp1 -d 192.168.202.47@tcp1 -m GET -e local_interrupt -r 1 [ 1276.076671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.147@tcp [ 1276.093588] LNet: There was an unexpected network error while writing to 192.168.202.147: rc = -22 [ 1276.097347] LNet: 1 local NIs in recovery (showing 1): 192.168.202.47@tcp [ 1276.395911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1276.431632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1276.456096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1276.483917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1276.517929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1276.538738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1276.578313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1276.634752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1276.649853] LNet: 60606:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1276.667812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1276.708642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1276.746884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1276.794644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1276.824405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1276.849836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1276.883048] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1277.493843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.47@tcp -d 192.168.202.147@tcp -m GET -e local_dropped -r 1 [ 1277.531388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.47@tcp -d 192.168.202.147@tcp1 -m GET -e local_dropped -r 1 [ 1277.565557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.47@tcp -d 192.168.202.47@tcp -m GET -e local_dropped -r 1 [ 1277.592626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.47@tcp -d 192.168.202.47@tcp1 -m GET -e local_dropped -r 1 [ 1277.628923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.47@tcp1 -d 192.168.202.147@tcp -m GET -e local_dropped -r 1 [ 1277.665190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.47@tcp1 -d 192.168.202.147@tcp1 -m GET -e local_dropped -r 1 [ 1277.694199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.47@tcp1 -d 192.168.202.47@tcp -m GET -e local_dropped -r 1 [ 1277.722473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.47@tcp1 -d 192.168.202.47@tcp1 -m GET -e local_dropped -r 1 [ 1277.760129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.147@tcp [ 1277.781821] LNet: There was an unexpected network error while writing to 192.168.202.147: rc = -22 [ 1277.786819] LNet: Skipped 2 previous similar messages [ 1278.010489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1278.034785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1278.062811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1278.091375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1278.116607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1278.150901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1278.186872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1278.215818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1278.234791] LNet: 60727:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1278.255161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1278.289561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1278.320652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1278.357555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1278.384878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1278.420120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1278.450199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1279.031873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.47@tcp -d 192.168.202.147@tcp -m GET -e local_aborted -r 1 [ 1279.082354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.47@tcp -d 192.168.202.147@tcp1 -m GET -e local_aborted -r 1 [ 1279.142065] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.47@tcp -d 192.168.202.47@tcp -m GET -e local_aborted -r 1 [ 1279.240792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.47@tcp -d 192.168.202.47@tcp1 -m GET -e local_aborted -r 1 [ 1279.296073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.47@tcp1 -d 192.168.202.147@tcp -m GET -e local_aborted -r 1 [ 1279.326588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.47@tcp1 -d 192.168.202.147@tcp1 -m GET -e local_aborted -r 1 [ 1279.367962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.47@tcp1 -d 192.168.202.47@tcp -m GET -e local_aborted -r 1 [ 1279.418104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.47@tcp1 -d 192.168.202.47@tcp1 -m GET -e local_aborted -r 1 [ 1279.462099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.147@tcp [ 1279.516457] LNet: There was an unexpected network error while writing to 192.168.202.147: rc = -22 [ 1279.519338] LNet: Skipped 2 previous similar messages [ 1279.785799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1279.813867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1279.837929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1279.876980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1279.904428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1279.939778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1279.975534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1280.017238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1280.033623] LNet: 60848:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1280.056184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1280.091392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1280.116531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1280.141872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1280.177738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1280.211907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1280.256682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1280.852463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.47@tcp -d 192.168.202.147@tcp -m GET -e local_no_route -r 1 [ 1280.880182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.47@tcp -d 192.168.202.147@tcp1 -m GET -e local_no_route -r 1 [ 1280.901795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.47@tcp -d 192.168.202.47@tcp -m GET -e local_no_route -r 1 [ 1280.926369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.47@tcp -d 192.168.202.47@tcp1 -m GET -e local_no_route -r 1 [ 1280.950767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.47@tcp1 -d 192.168.202.147@tcp -m GET -e local_no_route -r 1 [ 1280.982678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.47@tcp1 -d 192.168.202.147@tcp1 -m GET -e local_no_route -r 1 [ 1281.016284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.47@tcp1 -d 192.168.202.47@tcp -m GET -e local_no_route -r 1 [ 1281.054505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.47@tcp1 -d 192.168.202.47@tcp1 -m GET -e local_no_route -r 1 [ 1281.087909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.147@tcp [ 1281.383459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1281.418356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1281.468190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1281.493404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1281.521978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1281.562599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1281.592851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1281.647987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1281.681467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1281.703609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1281.733615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1281.784644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1281.821275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1281.845756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1281.866799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1282.446971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.47@tcp -d 192.168.202.147@tcp -m GET -e local_timeout -r 1 [ 1282.478051] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.47@tcp -d 192.168.202.147@tcp1 -m GET -e local_timeout -r 1 [ 1282.511732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.47@tcp -d 192.168.202.47@tcp -m GET -e local_timeout -r 1 [ 1282.568790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.47@tcp -d 192.168.202.47@tcp1 -m GET -e local_timeout -r 1 [ 1282.604073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.47@tcp1 -d 192.168.202.147@tcp -m GET -e local_timeout -r 1 [ 1282.669059] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.47@tcp1 -d 192.168.202.147@tcp1 -m GET -e local_timeout -r 1 [ 1282.709503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.47@tcp1 -d 192.168.202.47@tcp -m GET -e local_timeout -r 1 [ 1282.773839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.47@tcp1 -d 192.168.202.47@tcp1 -m GET -e local_timeout -r 1 [ 1282.846557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.147@tcp [ 1282.871442] LNet: There was an unexpected network error while writing to 192.168.202.147: rc = -22 [ 1282.876579] LNet: Skipped 5 previous similar messages [ 1283.163555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1283.200808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1283.227274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1283.259376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1283.316545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1283.373920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1283.457719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1283.493904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1283.509462] LNet: 61091:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1283.513748] LNet: 61091:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1283.526611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1283.556619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1283.589685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1283.624782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1283.645126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1283.675536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1283.711773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1284.257638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.47@tcp -d 192.168.202.147@tcp -m GET -e local_error -r 1 [ 1284.316426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.47@tcp -d 192.168.202.147@tcp1 -m GET -e local_error -r 1 [ 1284.361135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.47@tcp -d 192.168.202.47@tcp -m GET -e local_error -r 1 [ 1284.413523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.47@tcp -d 192.168.202.47@tcp1 -m GET -e local_error -r 1 [ 1284.459294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.47@tcp1 -d 192.168.202.147@tcp -m GET -e local_error -r 1 [ 1284.499112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.47@tcp1 -d 192.168.202.147@tcp1 -m GET -e local_error -r 1 [ 1284.539612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.47@tcp1 -d 192.168.202.47@tcp -m GET -e local_error -r 1 [ 1284.581203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.47@tcp1 -d 192.168.202.47@tcp1 -m GET -e local_error -r 1 [ 1284.608807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.147@tcp [ 1284.934079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1284.951654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1284.973602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1285.005465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1285.039447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1285.096765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1285.148274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1285.184426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1285.226795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1285.269435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1285.319072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1285.367861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1285.412418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1285.513991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1285.563665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1291.481546] LNet: 61292:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1291.486330] LNetError: 61292:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1291.492695] LNet: Removed LNI 192.168.202.47@tcp [ 1291.506576] LNet: Removed LNI 192.168.202.47@tcp1 [ 1291.975877] Key type .llcrypt unregistered [ 1291.977158] Key type ._llcrypt unregistered [ 1310.088049] Key type ._llcrypt registered [ 1310.093165] Key type .llcrypt registered [ 1311.269040] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 18:05:03 (1786831503) [ 1311.968385] Key type .llcrypt unregistered [ 1311.970527] Key type ._llcrypt unregistered [ 1318.735979] Key type ._llcrypt registered [ 1318.739623] Key type .llcrypt registered [ 1319.060853] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1319.104783] alg: No test for adler32 (adler32-zlib) [ 1320.044753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1320.092758] LNet: Added LNI 192.168.202.47@tcp [8/256/0/180] [ 1327.487274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.147@tcp [ 1328.257073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.47@tcp -d 192.168.202.147@tcp -m GET -e remote_dropped -r 1 [ 1328.295451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.47@tcp -d 192.168.202.47@tcp -m GET -e remote_dropped -r 1 [ 1328.331364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.147@tcp [ 1328.345651] LNet: There was an unexpected network error while writing to 192.168.202.147: rc = -22 [ 1328.522711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1328.543924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1328.569859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1328.595876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1328.623905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1328.665082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1328.715258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1328.755987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1328.773191] LNet: 62544:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1328.790929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1328.827812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1328.868480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1328.903695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1328.941474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1328.980872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1329.018307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1329.728292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.47@tcp -d 192.168.202.147@tcp -m GET -e remote_error -r 1 [ 1329.769048] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.47@tcp -d 192.168.202.47@tcp -m GET -e remote_error -r 1 [ 1329.801649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.147@tcp [ 1329.821652] LNet: There was an unexpected network error while writing to 192.168.202.147: rc = -22 [ 1330.001154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1330.017239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1330.031902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1330.049581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1330.067887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1330.081794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1330.096223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1330.110967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1330.119704] LNet: 62654:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1330.130532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1330.146937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1330.164386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1330.181862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1330.199483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1330.218213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1330.240745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1330.607078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.47@tcp -d 192.168.202.147@tcp -m GET -e remote_timeout -r 1 [ 1330.625678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.47@tcp -d 192.168.202.47@tcp -m GET -e remote_timeout -r 1 [ 1330.649774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.147@tcp [ 1330.834208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1330.868187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1330.916884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1330.930287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1330.944066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1330.973028] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1330.988353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1331.006179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1331.024709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1331.046988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1331.082124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1331.117789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1331.146192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1331.187632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1331.227054] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1331.494373] LNet: 62808:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1331.501035] LNetError: 62808:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1331.517405] LNet: Removed LNI 192.168.202.47@tcp [ 1331.808624] Key type .llcrypt unregistered [ 1331.810586] Key type ._llcrypt unregistered [ 1348.285726] Key type ._llcrypt registered [ 1348.289276] Key type .llcrypt registered [ 1349.771873] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 18:05:41 (1786831541) [ 1350.883483] Key type .llcrypt unregistered [ 1350.885933] Key type ._llcrypt unregistered [ 1357.320760] Key type ._llcrypt registered [ 1357.323904] Key type .llcrypt registered [ 1357.569552] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1357.599410] alg: No test for adler32 (adler32-zlib) [ 1358.563555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1358.603284] LNet: Added LNI 192.168.202.47@tcp [8/256/0/180] [ 1366.515393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.147@tcp [ 1377.437805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1377.529439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1377.580620] LNet: Added LNI 192.168.202.47@tcp1 [8/256/0/180] [ 1379.114581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.47@tcp -d 192.168.202.147@tcp -m GET -e remote_dropped -r 1 [ 1379.150638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.47@tcp -d 192.168.202.147@tcp1 -m GET -e remote_dropped -r 1 [ 1379.227651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.47@tcp -d 192.168.202.47@tcp -m GET -e remote_dropped -r 1 [ 1379.276850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.47@tcp -d 192.168.202.47@tcp1 -m GET -e remote_dropped -r 1 [ 1379.335550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.47@tcp1 -d 192.168.202.147@tcp -m GET -e remote_dropped -r 1 [ 1379.390236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.47@tcp1 -d 192.168.202.147@tcp1 -m GET -e remote_dropped -r 1 [ 1379.475562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.47@tcp1 -d 192.168.202.47@tcp -m GET -e remote_dropped -r 1 [ 1379.520262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.47@tcp1 -d 192.168.202.47@tcp1 -m GET -e remote_dropped -r 1 [ 1379.577438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.147@tcp [ 1379.617400] LNet: There was an unexpected network error while writing to 192.168.202.147: rc = -22 [ 1379.624588] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.147@tcp [ 1379.976029] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1380.022252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1380.063064] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1380.113519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1380.151304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1380.222222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1380.303927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1380.356787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1380.399810] LNet: 64279:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1380.442610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1380.549758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1380.603844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1380.644833] LNet: There was an unexpected network error while writing to 192.168.202.147: rc = -22 [ 1380.652567] LNet: Skipped 3 previous similar messages [ 1380.663490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1380.712431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1380.781111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1380.816442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1380.851773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1381.626063] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.47@tcp -d 192.168.202.147@tcp -m GET -e remote_error -r 1 [ 1381.680441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.47@tcp -d 192.168.202.147@tcp1 -m GET -e remote_error -r 1 [ 1381.739768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.47@tcp -d 192.168.202.47@tcp -m GET -e remote_error -r 1 [ 1381.775339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.47@tcp -d 192.168.202.47@tcp1 -m GET -e remote_error -r 1 [ 1381.816657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.47@tcp1 -d 192.168.202.147@tcp -m GET -e remote_error -r 1 [ 1381.868548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.47@tcp1 -d 192.168.202.147@tcp1 -m GET -e remote_error -r 1 [ 1381.926671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.47@tcp1 -d 192.168.202.47@tcp -m GET -e remote_error -r 1 [ 1382.007781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.47@tcp1 -d 192.168.202.47@tcp1 -m GET -e remote_error -r 1 [ 1382.067671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.147@tcp [ 1382.102529] LNet: There was an unexpected network error while writing to 192.168.202.147: rc = -22 [ 1382.433471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1382.471479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1382.500170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1382.527486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1382.552373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1382.611957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1382.638525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1382.660910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1382.700059] LNet: 64403:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1382.718402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1382.743894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1382.764169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1382.782619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1382.804204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1382.829185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1382.857547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1382.878433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1383.317826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.47@tcp -d 192.168.202.147@tcp -m GET -e remote_timeout -r 1 [ 1383.384690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.47@tcp -d 192.168.202.147@tcp1 -m GET -e remote_timeout -r 1 [ 1383.502284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.47@tcp -d 192.168.202.47@tcp -m GET -e remote_timeout -r 1 [ 1383.566426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.47@tcp -d 192.168.202.47@tcp1 -m GET -e remote_timeout -r 1 [ 1383.614806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.47@tcp1 -d 192.168.202.147@tcp -m GET -e remote_timeout -r 1 [ 1383.653422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.47@tcp1 -d 192.168.202.147@tcp1 -m GET -e remote_timeout -r 1 [ 1383.688118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.47@tcp1 -d 192.168.202.47@tcp -m GET -e remote_timeout -r 1 [ 1383.723700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.47@tcp1 -d 192.168.202.47@tcp1 -m GET -e remote_timeout -r 1 [ 1383.777177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.147@tcp [ 1384.011740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1384.030441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1384.052927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1384.086526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1384.115396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1384.134503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1384.164643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1384.195800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1384.218768] LNet: 64526:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1384.239652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1384.283356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1384.322522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1384.357607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1384.392680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1384.407779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1384.445813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1384.497347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1390.296780] LNet: 64608:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1390.318924] LNetError: 64608:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1390.336210] LNet: Removed LNI 192.168.202.47@tcp [ 1390.361425] LNet: Removed LNI 192.168.202.47@tcp1 [ 1391.010239] Key type .llcrypt unregistered [ 1391.014628] Key type ._llcrypt unregistered [ 1412.868718] Key type ._llcrypt registered [ 1412.869946] Key type .llcrypt registered [ 1414.545060] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 18:06:45 (1786831605) [ 1416.165913] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 1417.936580] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 18:06:49 (1786831609) [ 1418.837392] Key type .llcrypt unregistered [ 1418.840577] Key type ._llcrypt unregistered [ 1425.372100] Key type ._llcrypt registered [ 1425.374585] Key type .llcrypt registered [ 1425.555601] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1425.566940] alg: No test for adler32 (adler32-zlib) [ 1426.535668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1426.578774] LNet: Added LNI 192.168.202.47@tcp [8/256/0/180] [ 1433.402233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.147@tcp [ 1433.934955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.47@tcp -d 192.168.202.147@tcp -m GET -e network_timeout -r 1 [ 1433.952695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.47@tcp -d 192.168.202.47@tcp -m GET -e network_timeout -r 1 [ 1433.971138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.147@tcp [ 1433.980093] LNet: There was an unexpected network error while writing to 192.168.202.147: rc = -22 [ 1434.182965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1434.215494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1434.243604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1434.276618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1434.331120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1434.376585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1434.415736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1434.461590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1434.493692] LNet: 66159:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1434.531675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1434.592150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1434.634723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1434.667636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1434.704625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1434.761145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1434.783441] LNet: 1 local NIs in recovery (showing 1): 192.168.202.47@tcp [ 1434.797281] LNet: There was an unexpected network error while writing to 192.168.202.47: rc = -22 [ 1435.095489] LNet: 66202:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1435.104795] LNetError: 66202:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1436.156886] LNet: Removed LNI 192.168.202.47@tcp [ 1436.648784] Key type .llcrypt unregistered [ 1436.653189] Key type ._llcrypt unregistered [ 1458.978591] Key type ._llcrypt registered [ 1458.980023] Key type .llcrypt registered [ 1459.301416] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1459.316431] alg: No test for adler32 (adler32-zlib) [ 1460.222307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1460.243510] LNet: Added LNI 192.168.202.47@tcp [8/256/0/180] [ 1469.406237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.147@tcp [ 1478.349776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1478.391246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1478.424716] LNet: Added LNI 192.168.202.47@tcp1 [8/256/0/180] [ 1480.055706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.47@tcp -d 192.168.202.147@tcp -m GET -e network_timeout -r 1 [ 1480.098156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.47@tcp -d 192.168.202.147@tcp1 -m GET -e network_timeout -r 1 [ 1480.140271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.47@tcp -d 192.168.202.47@tcp -m GET -e network_timeout -r 1 [ 1480.182914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.47@tcp -d 192.168.202.47@tcp1 -m GET -e network_timeout -r 1 [ 1480.245672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.47@tcp1 -d 192.168.202.147@tcp -m GET -e network_timeout -r 1 [ 1480.327336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.47@tcp1 -d 192.168.202.147@tcp1 -m GET -e network_timeout -r 1 [ 1480.362816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.47@tcp1 -d 192.168.202.47@tcp -m GET -e network_timeout -r 1 [ 1480.410760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.47@tcp1 -d 192.168.202.47@tcp1 -m GET -e network_timeout -r 1 [ 1480.504923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.147@tcp [ 1480.541754] LNet: There was an unexpected network error while writing to 192.168.202.147: rc = -22 [ 1480.735182] LNet: 1 local NIs in recovery (showing 1): 192.168.202.47@tcp [ 1480.737841] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.147@tcp [ 1480.900336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1480.925932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1480.953900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1480.982386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1481.009988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1481.038907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1481.069288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1481.096894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1481.114823] LNet: 67088:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1481.140803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1481.171206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1481.206816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1481.236229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1481.290698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1481.339443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1487.453729] LNet: There was an unexpected network error while writing to 192.168.202.147: rc = -22 [ 1487.713438] LNet: 67166:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1487.717225] LNetError: 67166:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1487.724287] LNet: Removed LNI 192.168.202.47@tcp [ 1487.738978] LNet: Removed LNI 192.168.202.47@tcp1 [ 1488.093193] Key type .llcrypt unregistered [ 1488.095178] Key type ._llcrypt unregistered [ 1504.485186] Key type ._llcrypt registered [ 1504.487714] Key type .llcrypt registered [ 1505.915553] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 18:08:17 (1786831697) [ 1506.404550] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1506.420827] alg: No test for adler32 (adler32-zlib) [ 1507.364869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1507.437683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1507.484392] LNet: Added LNI 192.168.202.47@tcp [8/256/0/180] [ 1508.193931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1508.225384] LNet: Added LNI 192.168.202.47@tcp1 [8/256/0/180] [ 1509.177876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.47@tcp [ 1509.238661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1509.304883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1509.366105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1509.424339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1509.475960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1509.523475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.202.47@tcp [ 1509.548843] LNetError: 68267:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.202.47@tcp to recovery [ 1510.495731] LNet: 1 local NIs in recovery (showing 1): 192.168.202.47@tcp [ 1510.495967] LNet: There was an unexpected network error while writing to 192.168.202.47: rc = -22 [ 1512.545295] LNet: There was an unexpected network error while writing to 192.168.202.47: rc = -22 [ 1513.171381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1516.639725] LNet: There was an unexpected network error while writing to 192.168.202.47: rc = -22 [ 1516.992900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1517.053296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1517.109670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1517.133619] LNet: 68374:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1517.139916] LNetError: 68374:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1517.154081] LNet: Removed LNI 192.168.202.47@tcp [ 1517.165070] LNet: Removed LNI 192.168.202.47@tcp1 [ 1517.190267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1517.255638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1517.276077] LNet: Added LNI 192.168.202.47@tcp [8/256/0/180] [ 1518.374277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1518.401909] LNet: Added LNI 192.168.202.47@tcp1 [8/256/0/180] [ 1519.515939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.47@tcp [ 1519.563120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1519.634551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1519.692170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1519.731743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1519.769431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1519.807472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1519.853262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.202.47@tcp [ 1519.870947] LNetError: 68647:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.202.47@tcp to recovery [ 1520.287475] LNet: There was an unexpected network error while writing to 192.168.202.47: rc = -22 [ 1523.492973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1526.433778] LNet: There was an unexpected network error while writing to 192.168.202.47: rc = -22 [ 1526.447819] LNet: Skipped 1 previous similar message [ 1531.860851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1531.947471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1531.985660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1541.166436] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 18:08:52 (1786831732) [ 1541.709676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1541.758119] LNet: 69365:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1541.770447] LNetError: 69365:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1541.773269] LNetError: 69365:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1541.784380] LNet: Removed LNI 192.168.202.47@tcp [ 1541.797529] LNet: Removed LNI 192.168.202.47@tcp1 [ 1541.821903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1541.893410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1541.918170] LNet: Added LNI 192.168.202.47@tcp [8/256/0/180] [ 1542.604759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1542.630874] LNet: Added LNI 192.168.202.47@tcp1 [8/256/0/180] [ 1543.540512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.47@tcp [ 1543.633685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1543.679153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1543.730700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1543.778274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1543.828607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1543.883779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.202.47@tcp --health 0 [ 1544.072806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1544.928971] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.47@tcp [ 1544.929742] LNet: There was an unexpected network error while writing to 192.168.202.47: rc = -22 [ 1544.949673] LNet: Skipped 1 previous similar message [ 1558.772919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1558.970624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1559.006289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1559.070595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.202.47@tcp --health 500 [ 1559.200851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1566.247316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1566.348591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1566.381840] LNet: 69849:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1566.394764] LNetError: 69849:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1566.411946] LNetError: 69849:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1566.426366] LNet: Removed LNI 192.168.202.47@tcp [ 1566.463394] LNet: Removed LNI 192.168.202.47@tcp1 [ 1566.529984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1566.612176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1566.651282] LNet: Added LNI 192.168.202.47@tcp [8/256/0/180] [ 1567.647484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1567.673108] LNet: Added LNI 192.168.202.47@tcp1 [8/256/0/180] [ 1568.758126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.47@tcp [ 1568.820376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1568.884887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1568.930759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1568.962838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1569.009402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1569.047917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1569.101522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.202.47@tcp --health 0 [ 1570.655696] LNet: There was an unexpected network error while writing to 192.168.202.47: rc = -22 [ 1570.661418] LNet: Skipped 3 previous similar messages [ 1571.687980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1581.279852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1581.332331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1581.439723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1589.030417] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 18:09:40 (1786831780) [ 1589.717426] LNet: 70896:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1589.722870] LNetError: 70896:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1589.727645] LNetError: 70896:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1589.736281] LNet: Removed LNI 192.168.202.47@tcp [ 1589.754072] LNet: Removed LNI 192.168.202.47@tcp1 [ 1590.568853] Key type .llcrypt unregistered [ 1590.572770] Key type ._llcrypt unregistered [ 1599.248742] Key type ._llcrypt registered [ 1599.255203] Key type .llcrypt registered [ 1599.662329] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1599.718115] alg: No test for adler32 (adler32-zlib) [ 1600.722883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1600.768687] LNet: Added LNI 192.168.202.47@tcp [8/256/0/180] [ 1611.665757] Lustre: DEBUG MARKER: Initial discovery [ 1611.759453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.202.147@tcp [ 1614.262779] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1614.475729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1614.511692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.202.147@tcp [ 1614.550519] LNet: There was an unexpected network error while writing to 192.168.202.147: rc = -22 [ 1614.589421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1614.627583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.202.147@tcp -r 1 -m GET -l 3 [ 1614.676758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.202.147@tcp -r 1 -m GET -e local_error [ 1614.704695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.202.147@tcp -r 1 -m PUT -l 6 [ 1615.136563] LNet: 1 local NIs in recovery (showing 1): 192.168.202.47@tcp [ 1616.563435] Lustre: DEBUG MARKER: Force oleg247-client.virtnet to discover 192.168.202.147@tcp (in background) [ 1616.722605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.202.147@tcp [ 1618.417369] Lustre: DEBUG MARKER: Force oleg247-server to discover 192.168.202.47@tcp [ 1619.881374] LNet: There was an unexpected network error while writing to 192.168.202.147: rc = -22 [ 1621.776608] Lustre: DEBUG MARKER: Wait for 71669 [ 1623.519413] Lustre: DEBUG MARKER: Finished wait on 71669 [ 1623.675087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay del -a [ 1623.734023] LNet: There was an unexpected network error while writing to 192.168.202.147: rc = -22 [ 1623.757346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1624.054693] LNet: 72133:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1624.065242] LNetError: 72133:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1624.080726] LNet: Removed LNI 192.168.202.47@tcp [ 1624.551205] Key type .llcrypt unregistered [ 1624.554313] Key type ._llcrypt unregistered [ 1641.880397] Key type ._llcrypt registered [ 1641.883877] Key type .llcrypt registered [ 1643.422892] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 18:10:35 (1786831835) [ 1644.151621] Key type .llcrypt unregistered [ 1644.153371] Key type ._llcrypt unregistered [ 1652.617281] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1653.008539] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1653.039473] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1653.055141] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1655.544981] Key type ._llcrypt registered [ 1655.547579] Key type .llcrypt registered [ 1655.701706] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1655.717493] alg: No test for adler32 (adler32-zlib) [ 1656.765501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1656.857273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1656.896757] LNet: Added LNI 192.168.202.47@tcp [8/256/0/180] [ 1657.434611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1657.451365] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1666.147961] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 18:10:58 (1786831858) [ 1666.429485] LNet: 74216:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1666.435420] LNetError: 74216:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1666.442853] LNet: Removed LNI 10.1.2.3@tcp [ 1666.458243] LNet: Removed LNI 192.168.202.47@tcp [ 1667.067350] Key type .llcrypt unregistered [ 1667.072956] Key type ._llcrypt unregistered [ 1674.767819] Key type ._llcrypt registered [ 1674.772242] Key type .llcrypt registered [ 1675.139805] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1675.157372] alg: No test for adler32 (adler32-zlib) [ 1676.134087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1676.189436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1676.218248] LNet: Added LNI 192.168.202.47@tcp [8/256/0/180] [ 1677.253224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1677.286815] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1678.247157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.202.47@tcp [ 1679.327508] LNetError: Unexpected error -22 connecting to 192.168.202.47@tcp at host 192.168.202.47:988 [ 1685.190553] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 18:11:17 (1786831877) [ 1685.439758] LNet: 75353:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1685.443560] LNetError: 75353:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1686.497671] LNet: Removed LNI 10.1.2.3@tcp [ 1686.502161] LNetError: 75353:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1686.534242] LNet: Removed LNI 192.168.202.47@tcp [ 1687.158479] Key type .llcrypt unregistered [ 1687.164105] Key type ._llcrypt unregistered [ 1694.479693] Key type ._llcrypt registered [ 1694.481869] Key type .llcrypt registered [ 1694.715876] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1694.749260] alg: No test for adler32 (adler32-zlib) [ 1695.728255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1695.807041] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1695.836065] LNet: Added LNI 192.168.202.47@tcp1 [8/256/0/180] [ 1696.534784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1696.584509] LNet: Added LNI 192.168.202.47@tcp2 [8/256/0/180] [ 1697.481350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.202.47@tcp1 --nid 192.168.202.47@tcp2 [ 1702.140329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.47@tcp1 192.168.202.47@tcp2 [ 1702.161137] LNetError: 76192:0:(lib-move.c:2101:lnet_handle_find_routed_path()) Unable to send message from 192.168.202.47@tcp1 to 192.168.202.47@tcp2 - Route table may be misconfigured [ 1702.177405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.47@tcp2 192.168.202.47@tcp1 [ 1708.738698] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 18:11:40 (1786831900) [ 1709.001731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1709.017632] LNet: 76765:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1709.031907] LNetError: 76765:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1709.047374] LNet: Removed LNI 192.168.202.47@tcp1 [ 1710.126150] LNet: Removed LNI 192.168.202.47@tcp2 [ 1710.193363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1710.298678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1710.352953] LNet: Added LNI 192.168.202.47@tcp [8/256/0/180] [ 1711.233806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1711.251772] LNet: Added LNI 192.168.202.47@tcp1 [8/256/0/180] [ 1711.971246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.47@tcp [ 1711.994980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.47@tcp 192.168.202.47@tcp [ 1712.023595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.47@tcp1 192.168.202.47@tcp1 [ 1712.062574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.202.47@tcp -d 192.168.202.47@tcp -e network_timeout [ 1712.104968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.202.47@tcp -d 192.168.202.47@tcp1 -e network_timeout [ 1712.141628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.202.47@tcp1 -d 192.168.202.47@tcp -e network_timeout [ 1712.206840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.202.47@tcp1 -d 192.168.202.47@tcp1 -e network_timeout [ 1712.252650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.47@tcp [ 1712.275355] LNet: There was an unexpected network error while writing to 192.168.202.47: rc = -22 [ 1712.289075] LNet: 1 local NIs in recovery (showing 1): 192.168.202.47@tcp [ 1712.300360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1712.393665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1712.479709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1718.397466] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 18:11:50 (1786831910) [ 1718.545697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1718.579520] LNet: 77624:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1718.586057] LNetError: 77624:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1718.592096] LNetError: 77624:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1718.616151] LNet: Removed LNI 192.168.202.47@tcp [ 1719.653977] LNet: Removed LNI 192.168.202.47@tcp1 [ 1719.734162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1719.855561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1720.382546] LNet: 77669:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1720.743933] Key type .llcrypt unregistered [ 1720.748798] Key type ._llcrypt unregistered [ 1732.626275] Key type ._llcrypt registered [ 1732.628239] Key type .llcrypt registered [ 1734.184533] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 18:12:05 (1786831925) [ 1734.723789] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1734.737636] alg: No test for adler32 (adler32-zlib) [ 1735.719357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1737.315217] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1739.137754] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 18:12:10 (1786831930) [ 1739.232404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1739.268208] LNet: 78762:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1739.297081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1739.389434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1739.406660] LNet: Added LNI 192.168.202.47@tcp [8/256/0/180] [ 1739.996822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1740.008914] LNet: Added LNI 192.168.202.47@tcp1 [8/256/0/180] [ 1740.938182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.47@tcp [ 1740.979902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.47@tcp1 [ 1741.008506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.47@tcp1 [ 1747.539504] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 18:12:19 (1786831939) [ 1749.628664] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg247-server [ 1750.943350] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 18:12:23 (1786831943) [ 1752.905432] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg247-server [ 1754.793316] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 18:12:26 (1786831946) [ 1756.710315] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg247-server [ 1758.144583] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 18:12:30 (1786831950) [ 1760.027351] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg247-server [ 1761.679959] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 18:12:33 (1786831953) [ 1763.925914] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg247-server [ 1765.405061] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 18:12:37 (1786831957) [ 1767.026466] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg247-server [ 1768.622984] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 18:12:40 (1786831960) [ 1770.132233] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg247-server [ 1771.243259] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 18:12:43 (1786831963) [ 1773.060207] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg247-server [ 1775.245860] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 18:12:46 (1786831966) [ 1777.401881] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg247-server [ 1779.353670] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 18:12:51 (1786831971) [ 1779.609749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1779.632941] LNet: 82646:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1779.643683] LNetError: 82646:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1779.652192] LNet: Removed LNI 192.168.202.47@tcp [ 1780.717695] LNet: Removed LNI 192.168.202.47@tcp1 [ 1780.788018] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1780.899826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1780.928154] LNet: Added LNI 192.168.202.47@tcp [8/256/0/180] [ 1781.669320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1781.747297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.47@tcp [ 1781.898561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1781.914896] LNet: 82794:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1781.919808] LNetError: 82794:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1781.925094] LNetError: 82794:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1782.950052] LNet: Removed LNI 192.168.202.47@tcp [ 1782.967720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1783.035352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1783.067500] LNet: Added LNI 192.168.202.47@tcp [8/256/0/180] [ 1783.773167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.47@tcp [ 1783.938913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1783.958862] LNet: 82939:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1783.963555] LNetError: 82939:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1783.980917] LNet: Removed LNI 192.168.202.47@tcp [ 1783.997223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1784.048461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1784.070087] LNet: Added LNI 192.168.202.47@tcp [8/256/0/180] [ 1784.880956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1784.928434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.47@tcp [ 1785.084580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1786.152613] LNet: Removed LNI 192.168.202.47@tcp [ 1786.215305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1786.296699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1786.315114] LNet: Added LNI 192.168.202.47@tcp [8/256/0/180] [ 1787.206154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.47@tcp [ 1787.382272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1787.405057] LNetError: 83232:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1787.409804] LNetError: 83232:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1787.417796] LNet: Removed LNI 192.168.202.47@tcp [ 1787.458570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1787.513769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1787.530118] LNet: Added LNI 192.168.202.47@tcp [8/256/0/180] [ 1788.409863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1788.510628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.47@tcp [ 1788.720625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1788.770521] LNet: 83379:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1788.787505] LNet: 83379:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 1789.865407] LNet: Removed LNI 192.168.202.47@tcp [ 1789.907349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1789.981571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1790.007641] LNet: Added LNI 192.168.202.47@tcp [8/256/0/180] [ 1790.830338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.47@tcp [ 1791.003430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1792.063639] LNet: Removed LNI 192.168.202.47@tcp [ 1792.103957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1792.171649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1792.204927] LNet: Added LNI 192.168.202.47@tcp [8/256/0/180] [ 1793.067554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1793.156453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.47@tcp [ 1793.356392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1793.382233] LNetError: 83672:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1793.387017] LNetError: 83672:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1793.398507] LNet: Removed LNI 192.168.202.47@tcp [ 1793.411272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1793.467370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1793.513953] LNet: Added LNI 192.168.202.47@tcp [8/256/0/180] [ 1794.177510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.47@tcp [ 1794.405125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1795.493681] LNet: Removed LNI 192.168.202.47@tcp [ 1795.509130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1795.554584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1795.570970] LNet: Added LNI 192.168.202.47@tcp [8/256/0/180] [ 1796.254824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1796.321598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.47@tcp [ 1796.611423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1797.670409] LNet: Removed LNI 192.168.202.47@tcp [ 1797.732542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1797.799881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1797.829886] LNet: Added LNI 192.168.202.47@tcp [8/256/0/180] [ 1798.636697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.47@tcp [ 1798.844703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1798.879270] LNet: 84110:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1798.884601] LNet: 84110:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 1799.973915] LNet: Removed LNI 192.168.202.47@tcp [ 1800.006958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1800.074251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1800.093303] LNet: Added LNI 192.168.202.47@tcp [8/256/0/180] [ 1800.812865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1800.909957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.47@tcp [ 1802.281979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1802.316621] LNetError: 84267:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1802.322772] LNetError: 84267:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 1802.335040] LNet: Removed LNI 192.168.202.47@tcp [ 1802.349148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1802.388945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1802.412911] LNet: Added LNI 192.168.202.47@tcp [8/256/0/180] [ 1803.300586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.47@tcp [ 1803.585564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1804.648500] LNet: Removed LNI 192.168.202.47@tcp [ 1804.678562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1804.747171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1804.782502] LNet: Added LNI 192.168.202.47@tcp [8/256/0/180] [ 1805.563511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1805.645142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1805.678378] LNet: Removed LNI 192.168.202.47@tcp [ 1805.695819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1805.774505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1805.824367] LNet: Added LNI 192.168.202.47@tcp [8/256/0/180] [ 1807.084695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1807.212855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1807.256319] LNet: Removed LNI 192.168.202.47@tcp [ 1807.290599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1807.397308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 128 [ 1807.526799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1807.577797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1807.647589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 1000000000000000000000000 [ 1807.790452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1807.847544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1807.945089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer foo [ 1808.011921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1808.059203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1808.142430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 8 [ 1808.197649] LNet: Added LNI 192.168.202.47@tcp [8/256/0/180] [ 1809.338497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1809.378708] LNet: Removed LNI 192.168.202.47@tcp [ 1809.423346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1809.529758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1818.422521] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 18:13:29 (1786832009) [ 1818.656412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1818.735141] LNet: 85429:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1818.737721] LNet: 85429:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 8 previous similar messages [ 1818.799974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1818.898409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1818.934541] LNet: Added LNI 192.168.202.47@tcp231 [8/256/0/180] [ 1821.158366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1821.221071] LNetError: 85572:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1821.228797] LNetError: 85572:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 1821.261813] LNet: Removed LNI 192.168.202.47@tcp231 [ 1821.307277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1821.457465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1821.524519] LNet: Added LNI 192.168.202.47@tcp231 [8/256/0/180] [ 1821.547136] LNet: 85580:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1822.445651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1822.484899] LNet: Removed LNI 192.168.202.47@tcp231 [ 1822.514989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1822.566601] LNet: Added LNI 192.168.202.47@tcp231 [8/256/0/180] [ 1824.667803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1824.724308] LNet: Removed LNI 192.168.202.47@tcp231 [ 1824.761352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1824.875493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1824.897312] LNet: Added LNI 192.168.202.47@tcp231 [8/256/0/0] [ 1824.901339] LNet: 85733:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1825.816368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1825.868180] LNet: Removed LNI 192.168.202.47@tcp231 [ 1825.925854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1826.003192] LNet: Added LNI 192.168.202.47@tcp231 [8/256/0/0] [ 1828.396181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1828.426255] LNet: Removed LNI 192.168.202.47@tcp231 [ 1828.444335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1828.532691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1828.578645] LNet: Added LNI 192.168.202.47@tcp231 [8/256/0/60] [ 1828.589367] LNet: 85885:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1829.196944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1829.242711] LNet: Removed LNI 192.168.202.47@tcp231 [ 1829.281926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1829.327705] LNet: Added LNI 192.168.202.47@tcp231 [8/256/0/60] [ 1830.962202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1831.027704] LNet: Removed LNI 192.168.202.47@tcp231 [ 1831.079704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1831.210595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1831.245883] LNet: Added LNI 192.168.202.47@tcp231 [8/256/0/180] [ 1831.257827] LNet: 86036:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1831.575475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1831.642429] LNet: Removed LNI 192.168.202.47@tcp231 [ 1831.673389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1831.693046] LNet: Added LNI 192.168.202.47@tcp231 [8/256/0/180] [ 1841.179458] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 18:13:52 (1786832032) [ 1841.660963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1841.695783] LNet: Removed LNI 192.168.202.47@tcp231 [ 1841.722439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1841.792567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1841.810433] LNet: Added LNI 192.168.202.47@tcp232 [8/256/0/180] [ 1842.747485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1842.787681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1842.810327] LNet: Removed LNI 192.168.202.47@tcp232 [ 1842.838359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1842.867649] LNet: Added LNI 192.168.202.47@tcp232 [8/256/0/180] [ 1843.591768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1843.625918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1843.658852] LNet: Removed LNI 192.168.202.47@tcp232 [ 1843.684754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1843.703869] LNet: Added LNI 192.168.202.47@tcp232 [8/256/0/180] [ 1844.617764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1844.664489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1844.704821] LNet: Removed LNI 192.168.202.47@tcp232 [ 1844.742895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1844.793721] LNet: Added LNI 192.168.202.47@tcp232 [8/256/0/180] [ 1845.638318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1845.659725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1845.684200] LNet: Removed LNI 192.168.202.47@tcp232 [ 1845.702147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1853.705123] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 18:14:05 (1786832045) [ 1854.458616] LNet: 87877:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1854.465361] LNet: 87877:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 5 previous similar messages [ 1855.380709] Key type .llcrypt unregistered [ 1855.382577] Key type ._llcrypt unregistered [ 1863.337489] Key type ._llcrypt registered [ 1863.339576] Key type .llcrypt registered [ 1863.753174] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1863.788903] alg: No test for adler32 (adler32-zlib) [ 1864.829687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1864.854464] LNet: Added LNI 192.168.202.47@tcp [8/256/0/180] [ 1873.057044] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.147@tcp [ 1881.686867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1881.753109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1881.801119] LNet: Added LNI 192.168.202.47@tcp1 [8/256/0/180] [ 1883.343742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.47@tcp -d 192.168.202.147@tcp -m GET -e local_interrupt -r 2 [ 1883.381673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.47@tcp -d 192.168.202.147@tcp1 -m GET -e local_interrupt -r 2 [ 1883.418468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.47@tcp -d 192.168.202.47@tcp -m GET -e local_interrupt -r 2 [ 1883.456941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.47@tcp -d 192.168.202.47@tcp1 -m GET -e local_interrupt -r 2 [ 1883.493445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.47@tcp1 -d 192.168.202.147@tcp -m GET -e local_interrupt -r 2 [ 1883.539517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.47@tcp1 -d 192.168.202.147@tcp1 -m GET -e local_interrupt -r 2 [ 1883.571911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.47@tcp1 -d 192.168.202.47@tcp -m GET -e local_interrupt -r 2 [ 1883.614245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.47@tcp1 -d 192.168.202.47@tcp1 -m GET -e local_interrupt -r 2 [ 1883.656563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.47@tcp 192.168.202.147@tcp [ 1883.683435] LNet: There was an unexpected network error while writing to 192.168.202.147: rc = -22 [ 1883.692846] LNet: 1 local NIs in recovery (showing 1): 192.168.202.47@tcp [ 1883.984777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1884.037591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1884.085780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1884.133795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1884.176924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1884.228935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1884.275991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1884.320327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1884.345970] LNet: 88544:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1884.366107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1884.412518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1884.445466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1884.487200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1884.540888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1884.568830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1885.041650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.47@tcp -d 192.168.202.147@tcp -m GET -e local_dropped -r 2 [ 1885.060665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.47@tcp -d 192.168.202.147@tcp1 -m GET -e local_dropped -r 2 [ 1885.084717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.47@tcp -d 192.168.202.47@tcp -m GET -e local_dropped -r 2 [ 1885.126914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.47@tcp -d 192.168.202.47@tcp1 -m GET -e local_dropped -r 2 [ 1885.173871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.47@tcp1 -d 192.168.202.147@tcp -m GET -e local_dropped -r 2 [ 1885.221784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.47@tcp1 -d 192.168.202.147@tcp1 -m GET -e local_dropped -r 2 [ 1885.263959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.47@tcp1 -d 192.168.202.47@tcp -m GET -e local_dropped -r 2 [ 1885.346573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.47@tcp1 -d 192.168.202.47@tcp1 -m GET -e local_dropped -r 2 [ 1885.464062] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.47@tcp 192.168.202.147@tcp [ 1885.486686] LNet: There was an unexpected network error while writing to 192.168.202.147: rc = -22 [ 1885.732684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1885.773828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1885.810225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1885.842437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1885.884062] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1885.926272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1885.957277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1885.979357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1885.988864] LNet: 88665:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1886.003966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1886.028843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1886.060477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1886.095947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1886.132894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1886.184601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1886.712403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.47@tcp -d 192.168.202.147@tcp -m GET -e local_aborted -r 2 [ 1886.757158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.47@tcp -d 192.168.202.147@tcp1 -m GET -e local_aborted -r 2 [ 1886.814400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.47@tcp -d 192.168.202.47@tcp -m GET -e local_aborted -r 2 [ 1886.881452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.47@tcp -d 192.168.202.47@tcp1 -m GET -e local_aborted -r 2 [ 1886.921738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.47@tcp1 -d 192.168.202.147@tcp -m GET -e local_aborted -r 2 [ 1886.982902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.47@tcp1 -d 192.168.202.147@tcp1 -m GET -e local_aborted -r 2 [ 1887.029893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.47@tcp1 -d 192.168.202.47@tcp -m GET -e local_aborted -r 2 [ 1887.076528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.47@tcp1 -d 192.168.202.47@tcp1 -m GET -e local_aborted -r 2 [ 1887.120795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.47@tcp 192.168.202.147@tcp [ 1887.153885] LNet: There was an unexpected network error while writing to 192.168.202.147: rc = -22 [ 1887.452374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1887.493778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1887.535296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1887.578090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1887.627497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1887.662421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1887.710451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1887.768741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1887.789337] LNet: 88785:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1887.812220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1887.855666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1887.897121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1887.958935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1888.024558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1888.101349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1888.894178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.47@tcp -d 192.168.202.147@tcp -m GET -e local_no_route -r 2 [ 1888.937981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.47@tcp -d 192.168.202.147@tcp1 -m GET -e local_no_route -r 2 [ 1888.999730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.47@tcp -d 192.168.202.47@tcp -m GET -e local_no_route -r 2 [ 1889.071883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.47@tcp -d 192.168.202.47@tcp1 -m GET -e local_no_route -r 2 [ 1889.169374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.47@tcp1 -d 192.168.202.147@tcp -m GET -e local_no_route -r 2 [ 1889.260916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.47@tcp1 -d 192.168.202.147@tcp1 -m GET -e local_no_route -r 2 [ 1889.338038] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.47@tcp1 -d 192.168.202.47@tcp -m GET -e local_no_route -r 2 [ 1889.413324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.47@tcp1 -d 192.168.202.47@tcp1 -m GET -e local_no_route -r 2 [ 1889.492982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.47@tcp 192.168.202.147@tcp [ 1889.532723] LNet: There was an unexpected network error while writing to 192.168.202.147: rc = -22 [ 1889.931419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1889.976386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1890.028490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1890.106377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1890.178106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1890.234199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1890.302105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1890.357641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1890.386617] LNet: 88906:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1890.408277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1890.472775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1890.531968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1890.579459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1890.656605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1890.698635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1891.488134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.47@tcp -d 192.168.202.147@tcp -m GET -e local_timeout -r 2 [ 1891.556060] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.47@tcp -d 192.168.202.147@tcp1 -m GET -e local_timeout -r 2 [ 1891.595820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.47@tcp -d 192.168.202.47@tcp -m GET -e local_timeout -r 2 [ 1891.627270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.47@tcp -d 192.168.202.47@tcp1 -m GET -e local_timeout -r 2 [ 1891.658609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.47@tcp1 -d 192.168.202.147@tcp -m GET -e local_timeout -r 2 [ 1891.686501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.47@tcp1 -d 192.168.202.147@tcp1 -m GET -e local_timeout -r 2 [ 1891.713336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.47@tcp1 -d 192.168.202.47@tcp -m GET -e local_timeout -r 2 [ 1891.743443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.47@tcp1 -d 192.168.202.47@tcp1 -m GET -e local_timeout -r 2 [ 1891.778734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.47@tcp 192.168.202.147@tcp [ 1892.061525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1892.108989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1892.155674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1892.188664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1892.229325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1892.272402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1892.305595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1892.339707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1892.369693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1892.418327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1892.465317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1892.520631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1892.555409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1892.620353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1893.276334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.47@tcp -d 192.168.202.147@tcp -m GET -e remote_dropped -r 2 [ 1893.305610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.47@tcp -d 192.168.202.147@tcp1 -m GET -e remote_dropped -r 2 [ 1893.345622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.47@tcp -d 192.168.202.47@tcp -m GET -e remote_dropped -r 2 [ 1893.376244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.47@tcp -d 192.168.202.47@tcp1 -m GET -e remote_dropped -r 2 [ 1893.407723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.47@tcp1 -d 192.168.202.147@tcp -m GET -e remote_dropped -r 2 [ 1893.442665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.47@tcp1 -d 192.168.202.147@tcp1 -m GET -e remote_dropped -r 2 [ 1893.494606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.47@tcp1 -d 192.168.202.47@tcp -m GET -e remote_dropped -r 2 [ 1893.550853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.47@tcp1 -d 192.168.202.47@tcp1 -m GET -e remote_dropped -r 2 [ 1893.597978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.47@tcp 192.168.202.147@tcp [ 1893.633326] LNet: There was an unexpected network error while writing to 192.168.202.147: rc = -22 [ 1893.638499] LNet: Skipped 1 previous similar message [ 1893.640964] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.147@tcp [ 1893.899443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1893.934394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1893.970841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1893.999517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1894.030679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1894.067398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1894.110956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1894.153379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1894.197650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1894.246851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1894.305129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1894.355312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1894.410563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1894.442681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1899.980853] LNet: 89224:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1899.989559] LNetError: 89224:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1900.002701] LNet: Removed LNI 192.168.202.47@tcp [ 1900.015229] LNet: Removed LNI 192.168.202.47@tcp1 [ 1900.329892] Key type .llcrypt unregistered [ 1900.332974] Key type ._llcrypt unregistered [ 1917.918456] Key type ._llcrypt registered [ 1917.921598] Key type .llcrypt registered [ 1919.321494] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 18:15:11 (1786832111) [ 1919.727319] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1919.757799] alg: No test for adler32 (adler32-zlib) [ 1920.701137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1920.751672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1920.773434] LNet: Added LNI 192.168.202.47@tcp [8/256/0/180] [ 1921.560481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1921.605131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.202.47@tcp [ 1921.630617] LNetError: 90205:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.202.47@tcp to recovery [ 1921.739635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.202.47@tcp [ 1921.831339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1929.144716] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 18:15:21 (1786832121) [ 1929.309308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1929.356173] LNet: 90795:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1929.362890] LNetError: 90795:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1929.380307] LNet: Removed LNI 192.168.202.47@tcp [ 1929.409375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1929.486440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1929.536900] LNet: Added LNI 192.168.202.47@tcp [8/256/0/180] [ 1930.595054] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1930.613152] LNet: Added LNI 192.168.202.47@tcp1 [8/256/0/180] [ 1931.551715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.202.47@tcp [ 1931.580274] LNetError: 91061:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.202.47@tcp to recovery [ 1932.512993] LNet: 1 local NIs in recovery (showing 1): 192.168.202.47@tcp [ 1932.516593] LNet: There was an unexpected network error while writing to 192.168.202.47: rc = -22 [ 1932.825696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.202.47@tcp [ 1940.674400] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 18:15:32 (1786832132) [ 1942.237909] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg247-server [ 1943.475399] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 18:15:35 (1786832135) [ 1945.045883] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg247-server [ 1946.502084] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 18:15:38 (1786832138) [ 1947.763936] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg247-server [ 1948.854975] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 18:15:41 (1786832141) [ 1949.006450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1949.015561] LNet: 92690:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1949.019190] LNetError: 92690:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1949.024721] LNet: Removed LNI 192.168.202.47@tcp [ 1949.035973] LNet: Removed LNI 192.168.202.47@tcp1 [ 1949.060461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1949.091752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1949.120469] LNet: Added LNI 192.168.202.47@tcp [8/256/0/180] [ 1949.471576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1949.525148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1949.548790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1949.605123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1949.631327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1949.651842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1954.796282] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 18:15:47 (1786832147) [ 1955.007852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1955.052571] LNet: 93413:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1955.065591] LNetError: 93413:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1955.073951] LNetError: 93413:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1955.087829] LNet: Removed LNI 192.168.202.47@tcp [ 1955.106470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1955.161088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1955.179976] LNet: Added LNI 192.168.202.47@tcp [8/256/0/180] [ 1963.507472] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 18:15:55 (1786832155) [ 1963.689280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1963.716280] LNet: 94115:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1963.732889] LNetError: 94115:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1963.752725] LNet: Removed LNI 192.168.202.47@tcp [ 1963.797555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1963.896732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1963.920982] LNet: Added LNI 192.168.202.47@tcp [8/256/0/180] [ 1970.235925] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 18:16:02 (1786832162) [ 1971.502525] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 1973.103586] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 18:16:05 (1786832165) [ 1973.657876] LNet: 95154:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1973.661667] LNetError: 95154:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1973.673459] LNet: Removed LNI 192.168.202.47@tcp [ 1974.201622] Key type .llcrypt unregistered [ 1974.203917] Key type ._llcrypt unregistered [ 1981.800697] Key type ._llcrypt registered [ 1981.802463] Key type .llcrypt registered [ 1981.945688] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1981.961649] alg: No test for adler32 (adler32-zlib) [ 1983.005418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1983.067760] LNet: Added LNI 192.168.202.47@tcp [8/256/0/180] [ 1991.551219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.147@tcp [ 1997.437573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.202.147@tcp [ 1997.970839] LNet: 95581:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1997.990485] LNetError: 95581:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1998.029479] LNet: Removed LNI 192.168.202.47@tcp [ 1998.805619] Key type .llcrypt unregistered [ 1998.813808] Key type ._llcrypt unregistered [ 2015.207392] Key type ._llcrypt registered [ 2015.208974] Key type .llcrypt registered [ 2017.071877] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 2019.338539] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 2021.082950] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 18:16:53 (1786832213) [ 2022.533966] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 [ 2024.679889] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 18:16:56 (1786832216) [ 2027.155488] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg247-server [ 2029.055960] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 18:17:00 (1786832220) [ 2031.323338] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg247-server [ 2033.543670] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 18:17:05 (1786832225) [ 2035.224444] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2035.309411] alg: No test for adler32 (adler32-zlib) [ 2036.388483] Lustre: Lustre: Build Version: 2.17.57_3_g67c4c6e [ 2036.576310] LNet: Added LNI 192.168.202.47@tcp [8/256/0/180] [ 2038.247224] Key type lgssc registered [ 2039.380708] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2057.868986] Key type lgssc unregistered [ 2058.181735] LNet: 98761:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2058.187642] LNetError: 98761:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2058.200432] LNet: Removed LNI 192.168.202.47@tcp [ 2059.000701] Key type .llcrypt unregistered [ 2059.004520] Key type ._llcrypt unregistered [ 2074.562623] Key type ._llcrypt registered [ 2074.564267] Key type .llcrypt registered [ 2075.989385] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 18:17:47 (1786832267) [ 2076.389908] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2076.408138] alg: No test for adler32 (adler32-zlib) [ 2077.345459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2077.473288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2077.546585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2077.619914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2077.725463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2084.072345] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 18:17:56 (1786832276) [ 2084.305723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2084.321715] LNet: 100178:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2084.337675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2084.381710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2084.405895] LNet: Added LNI 192.168.202.47@tcp [8/256/0/180] [ 2085.271734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2085.300249] LNet: 100322:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2085.307877] LNetError: 100322:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2085.323262] LNet: Removed LNI 192.168.202.47@tcp [ 2085.348381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2085.404443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 2085.425385] LNet: Added LNI 192.168.202.47@tcp [8/256/0/180] [ 2085.858512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2085.886831] LNetError: 100350:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2085.906744] LNet: Removed LNI 192.168.202.47@tcp [ 2085.938486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2085.988874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 2086.014060] LNet: Added LNI 192.168.202.47@tcp [8/256/0/180] [ 2093.409089] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 18:18:05 (1786832285) [ 2093.567130] LNet: 100935:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2093.574442] LNet: 100935:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 2093.587124] LNetError: 100935:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2093.641525] LNet: Removed LNI 192.168.202.47@tcp [ 2094.291892] Key type .llcrypt unregistered [ 2094.294690] Key type ._llcrypt unregistered [ 2100.936813] Key type ._llcrypt registered [ 2100.939574] Key type .llcrypt registered [ 2101.170560] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2101.198763] alg: No test for adler32 (adler32-zlib) [ 2102.085562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2102.137769] LNet: Added LNI 192.168.202.47@tcp [8/256/0/180] [ 2102.140260] LNet: *** cfs_fail_loc=e003, val=100*** [ 2102.887768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.47@tcp [ 2102.900433] LNet: *** cfs_fail_loc=e003, val=100*** [ 2102.902355] LNet: Skipped 2 previous similar messages [ 2102.908411] Ping reply received with mapping: [ 2102.912753] found metadata reply NI[192.168.202.47@tcp] [ 2102.915442] 2f ca a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 2102.918384] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 2102.920887] 5a 5a 5a 5a 5a 5a 5a 5a [ 2103.093181] LNet: 101341:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2103.098463] LNetError: 101341:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2103.127866] LNet: Removed LNI 192.168.202.47@tcp [ 2103.702468] Key type .llcrypt unregistered [ 2103.708422] Key type ._llcrypt unregistered [ 2114.783980] Key type ._llcrypt registered [ 2114.785471] Key type .llcrypt registered [ 2116.374841] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 18:18:27 (1786832307) [ 2117.068176] Key type .llcrypt unregistered [ 2117.069784] Key type ._llcrypt unregistered [ 2123.541279] Key type ._llcrypt registered [ 2123.543101] Key type .llcrypt registered [ 2123.795473] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2123.809860] alg: No test for adler32 (adler32-zlib) [ 2124.918718] LNetError: 102400:0:(api-ni.c:2850:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 2125.197808] Key type .llcrypt unregistered [ 2125.200553] Key type ._llcrypt unregistered [ 2130.690498] Key type ._llcrypt registered [ 2130.696358] Key type .llcrypt registered [ 2132.096796] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 18:18:43 (1786832323) [ 2132.858310] Key type .llcrypt unregistered [ 2132.859912] Key type ._llcrypt unregistered [ 2140.311073] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2140.314631] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2142.954926] Key type ._llcrypt registered [ 2142.960176] Key type .llcrypt registered [ 2143.132548] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2143.143515] alg: No test for adler32 (adler32-zlib) [ 2144.047161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2144.110509] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2154.099895] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 18:19:06 (1786832346) [ 2154.265347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2154.291796] LNet: 104195:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2154.296458] LNetError: 104195:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2154.326877] LNet: Removed LNI 10.1.2.3@tcp [ 2154.360499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2155.111863] Key type .llcrypt unregistered [ 2155.114240] Key type ._llcrypt unregistered [ 2162.081522] Key type ._llcrypt registered [ 2162.084430] Key type .llcrypt registered [ 2162.312960] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2162.326849] alg: No test for adler32 (adler32-zlib) [ 2163.299064] LNet: Added LNI 192.168.202.47@tcp [8/256/0/180] [ 2163.369878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.47@tcp [ 2163.478932] LNet: 104490:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2163.484899] LNetError: 104490:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2164.522715] LNet: Removed LNI 192.168.202.47@tcp [ 2165.266662] Key type .llcrypt unregistered [ 2165.269671] Key type ._llcrypt unregistered [ 2177.166671] Key type ._llcrypt registered [ 2177.168283] Key type .llcrypt registered [ 2178.500160] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 18:19:30 (1786832370) [ 2179.366930] Key type .llcrypt unregistered [ 2179.370768] Key type ._llcrypt unregistered [ 2186.331609] Key type ._llcrypt registered [ 2186.337630] Key type .llcrypt registered [ 2186.482852] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2186.495534] alg: No test for adler32 (adler32-zlib) [ 2194.456318] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 18:19:45 (1786832385) [ 2194.574783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2194.671499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2194.762045] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2194.779907] LNet: Added LNI 192.168.202.47@tcp [8/256/0/180] [ 2195.419860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2195.484898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 2195.535320] LNet: Removed LNI 192.168.202.47@tcp [ 2195.926926] LNet: 106246:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2196.523794] Key type .llcrypt unregistered [ 2196.530858] Key type ._llcrypt unregistered [ 2210.143364] Key type ._llcrypt registered [ 2210.145784] Key type .llcrypt registered [ 2211.431269] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 18:20:03 (1786832403) [ 2211.797193] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2211.818309] alg: No test for adler32 (adler32-zlib) [ 2212.714709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2212.758914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2212.775665] LNet: Added LNI 192.168.202.47@tcp [8/256/0/180] [ 2213.417396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.47@tcp [ 2213.448404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.202.47@tcp [ 2213.462981] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 2213.466402] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 2213.469637] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2213, 0, 0 [ 2213.472953] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 2213.475868] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 2213.479452] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 2213.483728] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2213, 0, 0 [ 2213.488338] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 2213.491320] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2213.494954] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 2213.501453] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2213, 0, 2263 [ 2213.506414] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 2213.508392] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2213.511485] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 2213.513971] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2213, 0, 2263 [ 2213.519112] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 2219.733847] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 18:20:11 (1786832411) [ 2220.368869] LNet: 107786:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2220.376647] LNetError: 107786:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2220.383733] LNet: Removed LNI 192.168.202.47@tcp [ 2220.924401] Key type .llcrypt unregistered [ 2220.925966] Key type ._llcrypt unregistered [ 2228.344346] Key type ._llcrypt registered [ 2228.347581] Key type .llcrypt registered [ 2228.559685] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2228.608193] alg: No test for adler32 (adler32-zlib) [ 2229.580203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2229.613922] LNet: Added LNI 192.168.202.47@tcp [8/256/0/180] [ 2237.768818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.147@tcp [ 2247.749330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2247.822444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2247.844702] LNet: Added LNI 192.168.202.47@tcp1 [8/256/0/180] [ 2249.083339] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2249.257532] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2249.265752] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2249.275048] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2251.851669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 2251.889559] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 2263.746378] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 18:20:55 (1786832455) [ 2263.956243] LNet: 109294:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2263.963669] LNetError: 109294:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2263.969489] LNet: Removed LNI 192.168.202.47@tcp [ 2264.991368] LNet: Removed LNI 192.168.202.47@tcp1 [ 2264.997923] LNetError: 109294:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2265.004990] LNetError: 109294:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2265.029468] LNet: Removed LNI 10.1.2.3@tcp99 [ 2265.977316] Key type .llcrypt unregistered [ 2265.980839] Key type ._llcrypt unregistered [ 2272.255362] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2272.261407] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2274.891698] Key type ._llcrypt registered [ 2274.893658] Key type .llcrypt registered [ 2275.022305] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2275.034761] alg: No test for adler32 (adler32-zlib) [ 2275.950477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2276.010796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2276.043325] LNet: Added LNI 192.168.202.47@tcp [8/256/0/180] [ 2276.819426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2276.835033] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2278.852509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 2278.907596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2278.978418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 2278.999615] LNetError: 110049:0:(peer.c:2362:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 2279.056788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2279.149126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.47@tcp [ 2279.616306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2279.771889] LNet: 110069:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2279.784823] LNetError: 110069:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2279.798145] LNet: Removed LNI 10.1.2.3@tcp [ 2280.878904] LNet: Removed LNI 192.168.202.47@tcp [ 2281.706502] Key type .llcrypt unregistered [ 2281.709461] Key type ._llcrypt unregistered [ 2292.894508] Key type ._llcrypt registered [ 2292.896473] Key type .llcrypt registered [ 2294.122764] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 18:21:26 (1786832486) [ 2294.517853] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2294.531209] alg: No test for adler32 (adler32-zlib) [ 2295.465316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2295.523831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2295.551052] LNet: Added LNI 192.168.202.47@tcp [8/256/0/180] [ 2296.308407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.47@tcp [ 2296.380369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg247-client@tcp [ 2302.761744] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 18:21:34 (1786832494) [ 2302.834391] LNet: 111598:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2302.842379] LNetError: 111598:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2302.853969] LNet: Removed LNI 192.168.202.47@tcp [ 2303.417607] Key type .llcrypt unregistered [ 2303.418919] Key type ._llcrypt unregistered [ 2309.243188] Key type ._llcrypt registered [ 2309.244990] Key type .llcrypt registered [ 2309.395612] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2309.411987] alg: No test for adler32 (adler32-zlib) [ 2310.319729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 2310.337136] LNet: Added LNI 192.168.202.47@tcp [8/256/0/180] [ 2310.365708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 2316.167962] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 18:21:48 (1786832508) [ 2317.910280] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 2319.779805] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 18:21:51 (1786832511) [ 2321.551664] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 2323.741769] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 18:21:55 (1786832515) [ 2323.898764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2323.932053] LNet: 113023:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2323.936375] LNetError: 113023:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2323.951246] LNet: Removed LNI 192.168.202.47@tcp [ 2323.972902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2324.103900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2324.146395] LNet: Added LNI 192.168.202.47@tcp [8/256/0/180] [ 2325.463371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2325.589700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2332.897381] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 18:22:04 (1786832524) [ 2333.188907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2333.211522] LNet: 113738:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2333.217846] LNetError: 113738:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2333.234280] LNet: Removed LNI 192.168.202.47@tcp [ 2333.260502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2333.347957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 2333.411323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2334.125403] Key type .llcrypt unregistered [ 2334.132330] Key type ._llcrypt unregistered [ 2346.674549] Key type ._llcrypt registered [ 2346.676864] Key type .llcrypt registered [ 2347.999286] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 18:22:19 (1786832539) [ 2348.540199] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2348.553873] alg: No test for adler32 (adler32-zlib) [ 2349.466872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2349.542798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2349.575559] LNet: Added LNI 192.168.202.47@tcp [8/256/0/180] [ 2350.319147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2350.359326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.47@tcp [ 2356.904165] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 18:22:28 (1786832548) [ 2357.175801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2357.204655] LNet: 115301:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2357.212303] LNetError: 115301:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2358.248330] LNet: Removed LNI 192.168.202.47@tcp [ 2358.308679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2358.356851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 2358.382370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 2364.902655] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 18:22:36 (1786832556) [ 2365.078873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2365.099585] LNet: 115881:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2365.117637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2365.198140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2365.739037] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2365.792388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2365.855769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 2366.456156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2366.508711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2366.605739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2366.620799] LNet: 115914:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2366.643332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2366.688461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2367.141613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2367.208557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2367.303887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 2367.925787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2367.990847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2375.231822] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 18:22:47 (1786832567) [ 2375.410070] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2375.457384] LNet: 116514:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2375.481813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2375.551846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2375.582872] LNet: Added LNI 192.168.202.47@tcp [8/256/0/180] [ 2376.544370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2376.565759] LNet: Added LNI 192.168.202.47@tcp2 [8/256/0/180] [ 2377.554250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2377.629968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.47@tcp [ 2377.846203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2377.864434] LNetError: 116794:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2377.869312] LNet: Removed LNI 192.168.202.47@tcp [ 2377.877755] LNet: Removed LNI 192.168.202.47@tcp2 [ 2377.907895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2377.967189] LNet: Added LNI 192.168.202.47@tcp [8/256/0/180] [ 2377.973053] LNet: Added LNI 192.168.202.47@tcp2 [8/256/0/180] [ 2377.979527] LNet: 116800:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2378.436291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.47@tcp [ 2378.691842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2379.743283] LNet: Removed LNI 192.168.202.47@tcp [ 2379.752558] LNetError: 116837:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2379.772452] LNetError: 116837:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 2379.801963] LNet: Removed LNI 192.168.202.47@tcp2 [ 2379.854981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2380.040211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2380.087909] LNet: Added LNI 192.168.202.47@tcp [8/256/0/180] [ 2381.012678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2381.035394] LNet: Added LNI 192.168.202.47@tcp2 [8/256/0/180] [ 2381.963400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.47@tcp [ 2381.997732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2382.055993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.47@tcp [ 2382.285648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2382.314428] LNet: 117122:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2382.320458] LNet: 117122:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 2382.326962] LNetError: 117122:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2382.332465] LNet: Removed LNI 192.168.202.47@tcp [ 2383.401748] LNet: Removed LNI 192.168.202.47@tcp2 [ 2383.456419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2383.565168] LNet: Added LNI 192.168.202.47@tcp [8/256/0/180] [ 2383.573697] LNet: Added LNI 192.168.202.47@tcp2 [8/256/0/180] [ 2383.578369] LNet: 117128:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2384.477183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.47@tcp [ 2384.750423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2384.778706] LNet: Removed LNI 192.168.202.47@tcp [ 2385.844146] LNet: Removed LNI 192.168.202.47@tcp2 [ 2385.944899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2386.065290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2386.107980] LNet: Added LNI 192.168.202.47@tcp [8/256/0/180] [ 2386.950474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2386.980056] LNet: Added LNI 192.168.202.47@tcp2 [8/256/0/180] [ 2388.059608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2388.126365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.47@tcp [ 2388.376612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2388.392805] LNetError: 117444:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2388.397369] LNetError: 117444:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 2388.402433] LNet: Removed LNI 192.168.202.47@tcp [ 2388.412214] LNet: Removed LNI 192.168.202.47@tcp2 [ 2388.438253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2388.506108] LNet: Added LNI 192.168.202.47@tcp [8/256/0/180] [ 2388.514101] LNet: Added LNI 192.168.202.47@tcp2 [8/256/0/180] [ 2388.518970] LNet: 117451:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2389.113090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.47@tcp [ 2389.443261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2390.495368] LNet: Removed LNI 192.168.202.47@tcp [ 2390.515119] LNet: Removed LNI 192.168.202.47@tcp2 [ 2390.618802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2390.711127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2390.759240] LNet: Added LNI 192.168.202.47@tcp [8/256/0/180] [ 2391.602037] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2391.622963] LNet: Added LNI 192.168.202.47@tcp2 [8/256/0/180] [ 2392.434474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.47@tcp [ 2392.486487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2392.539209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.47@tcp [ 2392.807619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2392.836765] LNet: 117772:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2392.840583] LNet: 117772:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 2393.887316] LNet: Removed LNI 192.168.202.47@tcp [ 2394.918267] LNet: Removed LNI 192.168.202.47@tcp2 [ 2394.981437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2395.156136] LNet: Added LNI 192.168.202.47@tcp [8/256/0/180] [ 2395.169648] LNet: Added LNI 192.168.202.47@tcp2 [8/256/0/180] [ 2395.189546] LNet: 117778:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2395.908766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.47@tcp [ 2396.285994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2396.355941] LNet: Removed LNI 192.168.202.47@tcp [ 2397.421285] LNet: Removed LNI 192.168.202.47@tcp2 [ 2397.454532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2397.517297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2397.539247] LNet: Added LNI 192.168.202.47@tcp [8/256/0/180] [ 2398.277803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2398.296770] LNet: Added LNI 192.168.202.47@tcp2 [8/256/0/180] [ 2399.524031] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2399.564882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2399.638237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.47@tcp [ 2400.095915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2400.120406] LNetError: 118112:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2400.126861] LNetError: 118112:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 2400.133102] LNet: Removed LNI 192.168.202.47@tcp [ 2400.155412] LNet: Removed LNI 192.168.202.47@tcp2 [ 2400.174066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2400.243571] LNet: Added LNI 192.168.202.47@tcp [8/256/0/180] [ 2400.249643] LNet: Added LNI 192.168.202.47@tcp2 [8/256/0/180] [ 2400.254862] LNet: 118119:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2400.705873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.47@tcp [ 2401.007598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2402.082114] LNet: Removed LNI 192.168.202.47@tcp [ 2402.091048] LNet: Removed LNI 192.168.202.47@tcp2 [ 2402.110062] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2402.203945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2402.261674] LNet: Added LNI 192.168.202.47@tcp [8/256/0/180] [ 2403.515736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2403.563812] LNet: Added LNI 192.168.202.47@tcp2 [8/256/0/180] [ 2404.935181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.47@tcp [ 2404.974469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2405.006757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2405.073427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.47@tcp [ 2405.475282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2406.559227] LNet: Removed LNI 192.168.202.47@tcp [ 2407.589917] LNet: Removed LNI 192.168.202.47@tcp2 [ 2407.626194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2407.703573] LNet: Added LNI 192.168.202.47@tcp [8/256/0/180] [ 2407.711604] LNet: Added LNI 192.168.202.47@tcp2 [8/256/0/180] [ 2408.381164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.47@tcp [ 2415.523530] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 2417.318594] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 18:23:29 (1786832609) [ 2417.814982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2417.849315] LNet: 119239:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2417.858944] LNet: 119239:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 2417.885585] LNetError: 119239:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2417.901197] LNetError: 119239:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 2417.914125] LNet: Removed LNI 192.168.202.47@tcp [ 2418.989190] LNet: Removed LNI 192.168.202.47@tcp2 [ 2419.027703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2419.381837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2419.412133] LNet: Added LNI 192.168.202.47@tcp [8/256/0/180] [ 2420.160946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2420.182702] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2421.196185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2421.224736] LNet: Removed LNI 10.1.2.3@tcp [ 2421.238571] LNet: Removed LNI 192.168.202.47@tcp [ 2421.277269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2421.351413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 2421.397806] LNet: Added LNI 192.168.202.47@tcp [8/256/0/180] [ 2421.402750] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2422.009489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 2422.042362] LNet: Removed LNI 192.168.202.47@tcp [ 2422.068432] LNet: Removed LNI 10.1.2.3@tcp [ 2429.144055] Lustre: DEBUG MARKER: == sanity-lnet test 475: per-operation latency stats populate and reset ========================================================== 18:23:41 (1786832621) [ 2429.425803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2429.458318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2429.504650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2429.524108] LNet: Added LNI 192.168.202.47@tcp [8/256/0/180] [ 2430.328750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2430.402217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.47@tcp [ 2430.458898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.47@tcp [ 2430.517925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.47@tcp [ 2430.564192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.47@tcp [ 2430.601498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.47@tcp [ 2430.650135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.47@tcp [ 2430.767054] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.47@tcp [ 2430.845741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.47@tcp [ 2430.951227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.47@tcp [ 2431.003885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.47@tcp [ 2431.227547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2438.902781] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 18:23:50 (1786832630) [ 2439.199188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2440.301948] LNet: Removed LNI 192.168.202.47@tcp [ 2440.376165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2440.566693] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2440.722811] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2440.735119] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2440.742904] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2443.308858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2443.349975] LNet: Added LNI 192.168.202.47@tcp [8/256/0/180] [ 2443.994346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2444.018485] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2446.334794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.47@tcp [ 2446.368444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 2446.434675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.202.47@tcp -m PUT -e local_timeout -r 1 [ 2446.529461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 2446.638168] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2446.643129] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2446.995548] LNetError: 120920:0:(lib-move.c:4801:lnet_parse()) 192.168.202.47@tcp, src 192.168.202.47@tcp: Dropping PUT (error -108 looking up sender) [ 2448.034215] LNet: Removed LNI 10.1.2.3@tcp [ 2449.065832] LNet: Removed LNI 192.168.202.47@tcp [ 2449.509914] Key type .llcrypt unregistered [ 2449.521092] Key type ._llcrypt unregistered [ 2462.611219] Key type ._llcrypt registered [ 2462.612721] Key type .llcrypt registered [ 2463.876576] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 18:24:15 (1786832655) [ 2464.472347] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2464.485819] alg: No test for adler32 (adler32-zlib) [ 2465.487342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2465.995401] LNet: 122234:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2466.649452] Key type .llcrypt unregistered [ 2466.650929] Key type ._llcrypt unregistered [ 2473.423241] Key type ._llcrypt registered [ 2473.426946] Key type .llcrypt registered [ 2473.735972] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2473.767588] alg: No test for adler32 (adler32-zlib) [ 2474.752818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2474.775644] LNet: Added LNI 192.168.202.47@tcp [8/256/0/180] [ 2483.429135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.147@tcp [ 2483.668803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 2484.148427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.202.147@tcp [ 2484.352509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2484.376693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2484.400366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2484.443630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2484.513115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2484.547653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2484.629048] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2484.687632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2484.707390] LNet: 122688:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2484.730511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2484.756169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2484.794888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2484.825369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2484.855310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2484.890323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2485.685337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2485.953830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2485.981463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2486.009310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2486.050363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2486.055988] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.147@tcp [ 2486.094284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2486.177022] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2486.228263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2486.254717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2486.285265] LNet: 122805:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2486.308184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2486.354712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2486.403927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2486.443878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2486.475313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2486.524785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2487.198083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.202.147@tcp [ 2487.475747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2487.511638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2487.545780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2487.576777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2487.606162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2487.631261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2487.660706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2487.689919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2487.707045] LNet: 122922:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2487.729695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2487.762722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2487.808200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2487.834138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2487.865656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2487.889637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2488.500590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2488.747414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2488.776995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2488.815258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2488.839466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2488.862706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2488.889597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2488.925512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2488.954785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2488.980479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2489.010708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2489.041530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2489.075820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2489.109542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2489.145166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2489.449350] LNet: 123090:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2489.455565] LNetError: 123090:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2489.471833] LNet: Removed LNI 192.168.202.47@tcp [ 2489.814345] Key type .llcrypt unregistered [ 2489.819531] Key type ._llcrypt unregistered [ 2506.710935] Key type ._llcrypt registered [ 2506.714479] Key type .llcrypt registered [ 2508.146551] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 18:24:59 (1786832699) [ 2508.603339] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2508.628475] alg: No test for adler32 (adler32-zlib) [ 2509.691185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2510.228186] LNet: 123978:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2510.997448] Key type .llcrypt unregistered [ 2511.000615] Key type ._llcrypt unregistered [ 2517.847427] Key type ._llcrypt registered [ 2517.851882] Key type .llcrypt registered [ 2518.079667] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2518.090812] alg: No test for adler32 (adler32-zlib) [ 2519.058579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2519.082519] LNet: Added LNI 192.168.202.47@tcp [8/256/0/180] [ 2526.940273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.147@tcp [ 2536.593930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2536.640066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2536.672639] LNet: Added LNI 192.168.202.47@tcp1 [8/256/0/180] [ 2537.430804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 2538.189944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.202.147@tcp [ 2538.419102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2538.444312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2538.473541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2538.497723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2538.524933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2538.530885] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.147@tcp [ 2538.551677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2538.573966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2538.607151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2538.624771] LNet: 124635:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2538.641819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2538.684817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2538.745962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2538.795431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2538.827906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2538.871335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2539.429035] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2539.675566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2539.712782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2539.756063] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2539.785625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2539.824123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2539.863302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2539.919216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2539.955793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2539.975774] LNet: 124753:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2540.013327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2540.075678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2540.168314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2540.213663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2540.264833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2540.297893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2541.085950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.202.147@tcp [ 2541.303563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2541.356287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2541.403450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2541.452756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2541.514341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2541.559891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2541.631552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2541.676292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2541.701678] LNet: 124870:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2541.732348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2541.769661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2541.817081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2541.891681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2541.933706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2541.984761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2542.729772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2542.970350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2543.017285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2543.068978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2543.111618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2543.141872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2543.169985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2543.195198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2543.210507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2543.230925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2543.259715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2543.310753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2543.341035] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2543.398926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2543.449292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2548.959317] LNet: 125072:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2548.968723] LNetError: 125072:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2548.977603] LNet: Removed LNI 192.168.202.47@tcp [ 2548.995316] LNet: Removed LNI 192.168.202.47@tcp1 [ 2549.606645] Key type .llcrypt unregistered [ 2549.610177] Key type ._llcrypt unregistered [ 2571.679450] Key type ._llcrypt registered [ 2571.681560] Key type .llcrypt registered [ 2573.304724] Lustre: DEBUG MARKER: == sanity-lnet test 525: keep reachable peer on local net delete ========================================================== 18:26:04 (1786832764) [ 2573.734068] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2573.760028] alg: No test for adler32 (adler32-zlib) [ 2574.700329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2574.778074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2574.806681] LNet: Added LNI 192.168.202.47@tcp [8/256/0/180] [ 2575.572751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2575.591615] LNet: Added LNI 192.168.202.47@tcp2 [8/256/0/180] [ 2576.858282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.47@tcp [ 2576.989380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp [ 2578.079137] LNet: Removed LNI 192.168.202.47@tcp [ 2585.234379] Lustre: DEBUG MARKER: == sanity-lnet test 550: DD-off: keep route up when gw reachable via another local NI ========================================================== 18:26:16 (1786832776) [ 2587.544798] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_550 Need at least 2 local interfaces for multi-rail [ 2589.833357] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 18:26:21 (1786832781) [ 2598.496792] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 18:26:30 (1786832790) [ 2607.533727] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 18:26:38 (1786832798) [ 2616.834959] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 18:26:48 (1786832808) [ 2624.968559] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 18:26:56 (1786832816) [ 2631.563533] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 18:27:03 (1786832823) [ 2639.474800] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 18:27:11 (1786832831) [ 2647.882068] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 18:27:19 (1786832839) [ 2657.855746] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 18:27:29 (1786832849) [ 2666.253217] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 18:27:37 (1786832857) [ 2673.001170] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 18:27:45 (1786832865) [ 2679.823560] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 18:27:51 (1786832871) [ 2688.334796] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 18:28:00 (1786832880) [ 2695.792713] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 18:28:07 (1786832887) [ 2703.948612] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 18:28:15 (1786832895) [ 2713.751898] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 18:28:25 (1786832905) [ 2722.902877] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 18:28:34 (1786832914) [ 2731.933195] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 18:28:43 (1786832923) [ 2732.028332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2732.071858] LNet: 136765:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2732.094919] LNetError: 136765:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2732.126161] LNet: Removed LNI 192.168.202.47@tcp2 [ 2732.165963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2732.328863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2732.353359] LNet: Added LNI 192.168.202.47@tcp [8/256/0/180] [ 2733.700378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2733.740899] LNet: 136914:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2733.746700] LNetError: 136914:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2733.775938] LNet: Removed LNI 192.168.202.47@tcp [ 2733.821051] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2733.919145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2733.949376] LNet: Added LNI 192.168.202.47@tcp [8/256/0/180] [ 2742.136359] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 18:28:53 (1786832933) [ 2751.554320] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 18:29:03 (1786832943) [ 2760.554521] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 18:29:12 (1786832952) [ 2769.097556] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 18:29:20 (1786832960) [ 2777.718605] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 18:29:29 (1786832969) [ 2787.603849] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 18:29:39 (1786832979) [ 2794.814394] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 18:29:47 (1786832987) [ 2802.519855] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 18:29:54 (1786832994) [ 2809.233706] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 18:30:01 (1786833001) [ 2816.671812] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 18:30:08 (1786833008) [ 2823.875757] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 18:30:15 (1786833015) [ 2831.023720] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 18:30:22 (1786833022) [ 2838.787906] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 2564 sec ========== 18:30:30 (1786833030) [ 2839.193179] LNet: 144382:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2839.199980] LNetError: 144382:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2839.226329] LNet: Removed LNI 192.168.202.47@tcp [ 2840.031622] Key type .llcrypt unregistered [ 2840.033838] Key type ._llcrypt unregistered [ 2854.229651] Key type ._llcrypt registered [ 2854.275662] Key type .llcrypt registered [ 2854.577619] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2854.597614] alg: No test for adler32 (adler32-zlib) [ 2855.968990] Lustre: Lustre: Build Version: 2.17.57_3_g67c4c6e [ 2856.178774] LNet: Added LNI 192.168.202.47@tcp [8/256/0/180] [ 2857.839846] Key type lgssc registered [ 2859.106401] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2930.203069] Lustre: Mounted lustre-client - version 2.17.57_3_g67c4c6e [ 2935.244229] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2943.543461] Lustre: Unmounted lustre-client [ 2983.695767] Key type lgssc unregistered [ 2983.915213] LNet: 147345:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2983.931621] LNetError: 147345:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2985.013849] LNet: Removed LNI 192.168.202.47@tcp [ 2985.957340] Key type .llcrypt unregistered [ 2985.960826] Key type ._llcrypt unregistered