[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 430325352 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a2000-0x1465ccfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001010] APIC: Switch to symmetric I/O mode setup [ 0.002341] x2apic enabled [ 0.003008] Switched APIC routing to physical x2apic. [ 0.004012] kvm-guest: setup PV IPIs [ 0.006573] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007015] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008015] pid_max: default: 32768 minimum: 301 [ 0.009094] LSM: Security Framework initializing [ 0.010033] Yama: becoming mindful. [ 0.010486] SELinux: Initializing. [ 0.011045] *** VALIDATE selinux *** [ 0.018378] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.023431] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.025091] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.026089] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027100] *** VALIDATE tmpfs *** [ 0.028412] *** VALIDATE proc *** [ 0.030007] *** VALIDATE cgroup *** [ 0.030715] *** VALIDATE cgroup2 *** [ 0.031200] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.032096] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.033006] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.034028] Spectre V2 : User space: Vulnerable [ 0.035007] Speculative Store Bypass: Vulnerable [ 0.038218] debug: unmapping init [mem 0xffffffffa2c59000-0xffffffffa2c60fff] [ 0.040846] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.041719] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.042022] ... version: 2 [ 0.043018] ... bit width: 48 [ 0.044009] ... generic registers: 4 [ 0.044964] ... value mask: 0000ffffffffffff [ 0.045015] ... max period: 00007fffffffffff [ 0.046010] ... fixed-purpose events: 3 [ 0.046593] ... event mask: 000000070000000f [ 0.047350] rcu: Hierarchical SRCU implementation. [ 0.049364] smp: Bringing up secondary CPUs ... [ 0.050604] x86: Booting SMP configuration: [ 0.051019] .... node #0, CPUs: #1 #2 #3 [ 0.055624] smp: Brought up 1 node, 4 CPUs [ 0.056963] smpboot: Max logical packages: 1 [ 0.057015] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.187429] node 0 deferred pages initialised in 128ms [ 0.191167] devtmpfs: initialized [ 0.192231] x86/mm: Memory block size: 128MB [ 0.194859] gcov: version magic: 0x41383552 [ 0.196295] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.197070] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.198252] pinctrl core: initialized pinctrl subsystem [ 0.199180] [ 0.199577] ************************************************************* [ 0.200012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.201032] ** ** [ 0.202010] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.203023] ** ** [ 0.204013] ** This means that this kernel is built to expose internal ** [ 0.205012] ** IOMMU data structures, which may compromise security on ** [ 0.206014] ** your system. ** [ 0.207012] ** ** [ 0.208012] ** If you see this message and you are not debugging the ** [ 0.209010] ** kernel, report this immediately to your vendor! ** [ 0.210013] ** ** [ 0.211012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.212011] ************************************************************* [ 0.213682] NET: Registered protocol family 16 [ 0.214461] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.215057] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.216074] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.217439] cpuidle: using governor menu [ 0.219591] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.222446] PCI: Using configuration type 1 for base access [ 0.224133] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.235112] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.236019] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.238041] cryptd: max_cpu_qlen set to 1000 [ 0.240319] ACPI: Added _OSI(Module Device) [ 0.241013] ACPI: Added _OSI(Processor Device) [ 0.242016] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.243014] ACPI: Added _OSI(Processor Aggregator Device) [ 0.246273] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.249713] ACPI: Interpreter enabled [ 0.250069] ACPI: PM: (supports S0 S3 S4 S5) [ 0.251015] ACPI: Using IOAPIC for interrupt routing [ 0.252108] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.253389] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.263493] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.264040] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.265019] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.266073] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.268879] acpiphp: Slot [2] registered [ 0.269145] acpiphp: Slot [5] registered [ 0.270122] acpiphp: Slot [6] registered [ 0.271102] acpiphp: Slot [3] registered [ 0.272097] acpiphp: Slot [4] registered [ 0.273086] acpiphp: Slot [7] registered [ 0.274096] acpiphp: Slot [8] registered [ 0.275108] acpiphp: Slot [9] registered [ 0.276093] acpiphp: Slot [10] registered [ 0.278072] acpiphp: Slot [11] registered [ 0.279074] acpiphp: Slot [12] registered [ 0.280074] acpiphp: Slot [13] registered [ 0.281105] acpiphp: Slot [14] registered [ 0.282075] acpiphp: Slot [15] registered [ 0.283071] acpiphp: Slot [16] registered [ 0.284079] acpiphp: Slot [17] registered [ 0.285072] acpiphp: Slot [18] registered [ 0.285996] acpiphp: Slot [19] registered [ 0.287096] acpiphp: Slot [20] registered [ 0.288109] acpiphp: Slot [21] registered [ 0.290062] acpiphp: Slot [22] registered [ 0.291104] acpiphp: Slot [23] registered [ 0.292109] acpiphp: Slot [24] registered [ 0.293225] acpiphp: Slot [25] registered [ 0.294095] acpiphp: Slot [26] registered [ 0.296094] acpiphp: Slot [27] registered [ 0.297098] acpiphp: Slot [28] registered [ 0.299103] acpiphp: Slot [29] registered [ 0.301105] acpiphp: Slot [30] registered [ 0.302072] acpiphp: Slot [31] registered [ 0.303064] PCI host bridge to bus 0000:00 [ 0.304011] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.306013] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.307014] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.309038] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.312024] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.315037] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.317176] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.321025] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.324109] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.331706] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.336023] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.340025] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.343015] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.345016] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.348597] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.351226] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.354043] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.356717] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.361010] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.372014] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.376012] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.380733] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.389011] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.394930] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.412023] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.422856] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.428014] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.437011] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.452010] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.466328] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.468394] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.471488] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.474387] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.477287] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.482123] iommu: Default domain type: Passthrough [ 0.483469] SCSI subsystem initialized [ 0.485161] ACPI: bus type USB registered [ 0.487138] usbcore: registered new interface driver usbfs [ 0.489112] usbcore: registered new interface driver hub [ 0.491093] usbcore: registered new device driver usb [ 0.493209] pps_core: LinuxPPS API ver. 1 registered [ 0.495014] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.497095] PTP clock support registered [ 0.500043] EDAC MC: Ver: 3.0.0 [ 0.501490] PCI: Using ACPI for IRQ routing [ 0.503891] NetLabel: Initializing [ 0.505013] NetLabel: domain hash size = 128 [ 0.506011] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.509107] NetLabel: unlabeled traffic allowed by default [ 0.511157] vgaarb: loaded [ 0.512289] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.514021] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.521000] clocksource: Switched to clocksource kvm-clock [ 0.621156] VFS: Disk quotas dquot_6.6.0 [ 0.622654] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.624768] *** VALIDATE ramfs *** [ 0.626229] *** VALIDATE hugetlbfs *** [ 0.627636] pnp: PnP ACPI init [ 0.629892] pnp: PnP ACPI: found 6 devices [ 0.651112] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.653263] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.655184] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.657621] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.660262] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.663629] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.666164] NET: Registered protocol family 2 [ 0.668500] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.674098] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.678860] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.684473] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.688076] TCP: Hash tables configured (established 65536 bind 65536) [ 0.691234] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.694628] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.697610] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.700390] NET: Registered protocol family 1 [ 0.704515] RPC: Registered named UNIX socket transport module. [ 0.706582] RPC: Registered udp transport module. [ 0.707781] RPC: Registered tcp transport module. [ 0.708800] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.710692] NET: Registered protocol family 44 [ 0.711804] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.714283] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.716601] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.719208] PCI: CLS 0 bytes, default 64 [ 0.720910] Unpacking initramfs... [ 2.097460] debug: unmapping init [mem 0xffff91ffbcc64000-0xffff91ffbffcffff] [ 2.101034] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.103426] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.106217] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.632692] Initialise system trusted keyrings [ 2.635198] Key type blacklist registered [ 2.637655] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.647212] zbud: loaded [ 2.650634] *** VALIDATE nfs *** [ 2.652154] *** VALIDATE nfs4 *** [ 2.654239] pstore: using deflate compression [ 2.658272] Platform Keyring initialized [ 2.789169] NET: Registered protocol family 38 [ 2.790956] Key type asymmetric registered [ 2.792193] Asymmetric key parser 'x509' registered [ 2.795170] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.798093] io scheduler mq-deadline registered [ 2.799871] io scheduler kyber registered [ 2.801351] io scheduler bfq registered [ 2.802941] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.806456] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.809746] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.813215] ACPI: Power Button [PWRF] [ 2.818671] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.826444] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.841797] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.870918] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.900135] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.904319] Non-volatile memory driver v1.3 [ 2.905976] Linux agpgart interface v0.103 [ 2.944832] virtio_blk virtio1: [vda] 146680 512-byte logical blocks (75.1 MB/71.6 MiB) [ 2.949078] vda: detected capacity change from 0 to 75100160 [ 2.967403] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 2.970690] vdb: detected capacity change from 0 to 1073741824 [ 2.980259] libphy: Fixed MDIO Bus: probed [ 2.993471] usbcore: registered new interface driver usbserial_generic [ 2.996234] usbserial: USB Serial support registered for generic [ 2.998866] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.004250] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.006344] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.009325] mousedev: PS/2 mouse device common for all mice [ 3.013545] rtc_cmos 00:05: RTC can wake from S4 [ 3.016798] rtc_cmos 00:05: registered as rtc0 [ 3.018622] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.018972] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.019039] intel_pstate: CPU model not supported [ 3.024862] hid: raw HID events driver (C) Jiri Kosina [ 3.033420] usbcore: registered new interface driver usbhid [ 3.036175] usbhid: USB HID core driver [ 3.036961] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.038576] drop_monitor: Initializing network drop monitor service [ 3.045067] Initializing XFRM netlink socket [ 3.047285] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.047569] NET: Registered protocol family 10 [ 3.054499] Segment Routing with IPv6 [ 3.056453] NET: Registered protocol family 17 [ 3.059943] mpls_gso: MPLS GSO support [ 3.065397] RAS: Correctable Errors collector initialized. [ 3.067502] AVX version of gcm_enc/dec engaged. [ 3.069514] AES CTR mode by8 optimization enabled [ 3.157732] sched_clock: Marking stable (3157704928, 0)->(4057410535, -899705607) [ 3.162198] registered taskstats version 1 [ 3.164409] Loading compiled-in X.509 certificates [ 3.166538] zswap: loaded using pool lzo/zbud [ 3.196027] Key type big_key registered [ 3.210727] Key type encrypted registered [ 3.212764] ima: No TPM chip found, activating TPM-bypass! [ 3.215512] ima: Allocated hash algorithm: sha1 [ 3.217440] ima: No architecture policies found [ 3.219325] evm: Initialising EVM extended attributes: [ 3.221319] evm: security.selinux [ 3.222607] evm: security.ima [ 3.223638] evm: security.capability [ 3.224969] evm: HMAC attrs: 0x1 [ 3.227316] rtc_cmos 00:05: setting system clock to 2026-09-05 04:15:33 UTC (1788581733) [ 3.233640] debug: unmapping init [mem 0xffffffffa3c03000-0xffffffffa3dfffff] [ 3.236894] debug: unmapping init [mem 0xffffffffa2982000-0xffffffffa2c58fff] [ 3.246084] Write protecting the kernel read-only data: 28672k [ 3.250251] debug: unmapping init [mem 0xffffffffa1003000-0xffffffffa11fffff] [ 3.253464] debug: unmapping init [mem 0xffffffffa1914000-0xffffffffa19fffff] [ 3.285168] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.292806] systemd[1]: Detected virtualization kvm. [ 3.294630] systemd[1]: Detected architecture x86-64. [ 3.296433] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.321519] systemd[1]: No hostname configured. [ 3.323482] systemd[1]: Set hostname to . [ 3.325774] random: systemd: uninitialized urandom read (16 bytes read) [ 3.328752] systemd[1]: Initializing machine ID from random generator. [ 3.362843] random: ln: uninitialized urandom read (6 bytes read) [ 3.483262] random: systemd: uninitialized urandom read (16 bytes read) [ 3.486137] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 3.492171] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 3.497488] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Swap. Starting Create list of required st…ce nodes for the current kernel... Starting Setup Virtual Console... [ OK ] Listening on udev Control Socket. Starting Apply Kernel Variables... [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Timers. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Sockets. Starting Journal Service... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 5.387496] device-mapper: uevent: version 1.0.3 [ 5.389922] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 7.108321] virtio_net virtio0 ens2: renamed from eth0 [ 7.215996] scsi host0: ata_piix [ 7.238984] scsi host1: ata_piix [ 7.271661] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 7.279604] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 7.368452] random: fast init done [ 12.739169] random: crng init done [ 12.744551] random: 7 urandom warning(s) missed due to ratelimiting [ 13.510885] dracut-initqueue[591]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 15.335317] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 18.217867] printk: systemd: 26 output lines suppressed due to ratelimiting [ 19.009131] SELinux: Disabled at runtime. [ 19.138297] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 19.162101] systemd[1]: Detected virtualization kvm. [ 19.165501] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 20.791502] systemd[1]: initrd-switch-root.service: Succeeded. [ 20.802449] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 20.838249] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 20.863802] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 20.873875] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 20.908960] systemd[1]: Starting Journal Service... Starting Journal Service... [ 20.943088] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... Starting Remount Root and Kernel File Systems... Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Created slice User and Session Slice. [ OK ] Created slice system-getty.slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Stopped target Switch Root. [ 21.283075] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Stopped target Initrd Root File System. Mounting POSIX Message Queue File System... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on initctl Compatibility Named Pipe. Starting Create list of required st…ce nodes for the current kernel... Starting Apply Kernel Variables... Mounting Huge Pages File System... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Listening on Process Core Dump Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Reached target rpc_pipefs.target. Mounting Kernel Debug File System... [ OK ] Reached target Slices. [ OK ] Stopped target Initrd File Systems. [ OK ] Started Journal Service. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 22.599702] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 23.889996] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 24.079817] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 24.567348] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 24.646115] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (8s / no limit)[ 28.951531] Key type dns_resolver registered [** ] A start job is running for Configur…-only root support (8s / no limit) [*** ] A start job is running for Configur…-only root support (9s / no limit)[ 29.779503] NFS: Registering the id_resolver key type [ 29.785407] Key type id_resolver registered [ 29.787196] Key type id_legacy registered [ *** ] A start job is running for Configur…-only root support (9s / no limit) [ *** ] A start job is running for Configur…-only root support (9s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Login Service... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Started OpenSSH server daemon. Starting Hostname Service... [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg601-client login: [ 83.478773] libcfs: loading out-of-tree module taints kernel. [ 83.771834] Key type ._llcrypt registered [ 83.777536] Key type .llcrypt registered [ 84.258864] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 84.282223] alg: No test for adler32 (adler32-zlib) [ 85.702634] Lustre: Lustre: Build Version: 2.17.57_104_g4a8f637 [ 86.395275] LNet: Added LNI 192.168.206.1@tcp [8/256/0/180] [ 88.128137] Key type lgssc registered [ 89.841378] Lustre: Echo OBD driver; http://www.lustre.org/ [ 194.735703] Lustre: Mounted lustre-client - version 2.17.57_104_g4a8f637 [ 199.287609] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 206.251532] hrtimer: interrupt took 2715493 ns [ 211.710589] Lustre: DEBUG MARKER: oleg601-client.virtnet: executing check_logdir /tmp/testlogs/ [ 215.828432] Lustre: DEBUG MARKER: oleg601-client.virtnet: executing yml_node [ 219.726426] Lustre: DEBUG MARKER: Client: 2.17.57.104 [ 220.641139] Lustre: lustre-OST0000-osc-ffff920018281000: disconnect after 23s idle [ 221.990187] Lustre: DEBUG MARKER: MDS: 2.17.57.104 [ 224.155101] Lustre: DEBUG MARKER: OSS: 2.17.57.104 [ 225.395268] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Sat Sep 5 00:19:14 EDT 2026 [ 238.512862] Lustre: Unmounted lustre-client [ 254.475347] Key type lgssc unregistered [ 254.741868] LNet: 8433:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 254.749491] LNetError: 8433:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 254.771967] LNet: Removed LNI 192.168.206.1@tcp [ 255.494155] Key type .llcrypt unregistered [ 255.496269] Key type ._llcrypt unregistered [ 276.149099] Key type ._llcrypt registered [ 276.154343] Key type .llcrypt registered [ 276.300300] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 276.313731] alg: No test for adler32 (adler32-zlib) [ 277.222324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 277.294940] LNet: Added LNI 192.168.206.1@tcp [8/256/0/180] [ 277.310600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 280.937242] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 281.272729] LNet: 9312:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 281.283830] LNetError: 9312:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 281.294278] LNet: Removed LNI 192.168.206.1@tcp [ 281.895485] Key type .llcrypt unregistered [ 281.898732] Key type ._llcrypt unregistered [ 286.846605] Key type ._llcrypt registered [ 286.853449] Key type .llcrypt registered [ 287.725755] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 00:20:16 (1788582016) [ 287.923891] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 287.932725] alg: No test for adler32 (adler32-zlib) [ 288.815079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 288.883592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 288.899294] LNet: 9719:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 288.918320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 288.967940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 288.984711] LNet: 9726:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 294.144759] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 00:20:23 (1788582023) [ 294.251513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 294.268688] LNet: 10301:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 294.298870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 294.352413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 299.983668] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 00:20:29 (1788582029) [ 300.084127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 300.108794] LNet: 10882:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 300.134804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 300.232592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 305.606347] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 00:20:34 (1788582034) [ 305.705061] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 305.742215] LNet: 11465:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 305.765309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 305.849221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 310.779665] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 00:20:40 (1788582040) [ 310.903230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 310.963914] LNet: 12054:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 310.993399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 311.066869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 311.474086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 316.633622] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 00:20:45 (1788582045) [ 316.915573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 316.956094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 317.021611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 317.368295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 317.676396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 322.508500] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 00:20:51 (1788582051) [ 322.654321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 322.701535] LNet: 13241:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 322.710509] LNet: 13241:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 322.756362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 322.833615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 327.598787] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 00:20:56 (1788582056) [ 327.669542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 327.697740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 327.736806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 327.760906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 328.007714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 328.035903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 328.345942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 328.376644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 328.747563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 328.801338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 329.186916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 329.237795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 329.643647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 329.687138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 329.994242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 330.038622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 330.404228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 330.436382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 330.825821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 330.856142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 331.194685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 331.241638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 331.550476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 331.586944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 331.908232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 331.946120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 332.248327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 332.280526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 337.188874] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 00:21:06 (1788582066) [ 337.258672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 337.304976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 337.377855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 337.405391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 342.354465] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 00:21:11 (1788582071) [ 342.418906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 342.438134] LNet: 15090:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 342.444276] LNet: 15090:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 342.463183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 342.529482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 342.563646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 347.281919] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 00:21:16 (1788582076) [ 347.415567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 347.440569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 347.496799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 347.548495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 353.037088] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 00:21:22 (1788582082) [ 353.102314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 353.143141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 353.209871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 353.234775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 359.029931] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 00:21:28 (1788582088) [ 359.249802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 359.285442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 359.342240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 359.378116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 365.384846] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 00:21:34 (1788582094) [ 365.625861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 365.658755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 365.758164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 365.812883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 371.032383] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 00:21:40 (1788582100) [ 371.197189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 371.252405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 371.379260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 371.751942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 378.006103] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 00:21:47 (1788582107) [ 378.068174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 378.087520] LNet: 18619:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 378.093091] LNet: 18619:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 5 previous similar messages [ 378.125160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 378.264398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 378.517817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 383.793871] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 00:21:53 (1788582113) [ 383.891952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 383.919207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 384.029524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 384.189380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 389.819650] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 00:21:58 (1788582118) [ 389.970806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 390.033779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 390.462896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 390.732681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 396.160924] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 00:22:05 (1788582125) [ 396.278869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 396.345314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 396.395481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 396.427893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 402.235397] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 00:22:11 (1788582131) [ 402.458796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 402.511273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 402.638176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 402.720730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 409.151464] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 00:22:18 (1788582138) [ 409.361391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 409.392049] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 409.479707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 416.516687] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 00:22:25 (1788582145) [ 416.660724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 416.723779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 416.884865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 423.566166] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 00:22:32 (1788582152) [ 423.745633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 423.802289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 423.939325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 424.603226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 430.637817] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 00:22:39 (1788582159) [ 430.876647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 430.907680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 430.960549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 430.985846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 436.706551] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 00:22:45 (1788582165) [ 436.816119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 436.879333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 436.986172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 437.020816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 442.099319] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 00:22:51 (1788582171) [ 442.219400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 442.244732] LNet: 24637:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 442.249458] LNet: 24637:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 9 previous similar messages [ 442.285412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 442.348753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 442.376682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 447.391468] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 00:22:56 (1788582176) [ 447.590302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 447.619686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 447.682506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 447.740456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 453.786161] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 00:23:03 (1788582183) [ 453.973729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 454.004333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 454.040811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 454.087955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 454.105689] LNetError: 25826:0:(peer.c:2362:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 454.134890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 454.164974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 459.022870] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 00:23:08 (1788582188) [ 459.123726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 459.155692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 459.229857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.QinR7zsiL3 [ 459.303202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.QinR7zsiL3 [ 459.369798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.QinR7zsiL3 [ 464.905343] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 00:23:13 (1788582193) [ 464.989544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 465.033527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 465.084968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 465.111065] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 465.146837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 470.575782] Lustre: DEBUG MARKER: == sanity-lnet test 40: LNetAddPeer merges a pre-existing non-primary peer NID ========================================================== 00:23:19 (1788582199) [ 470.836405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 470.866847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 470.941540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 470.961097] LNet: Added LNI 192.168.206.1@tcp [8/256/0/180] [ 472.149630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.1@tcp [ 473.330315] Lustre: Lustre: Build Version: 2.17.57_104_g4a8f637 [ 475.248166] Key type lgssc registered [ 476.237139] Lustre: Echo OBD driver; http://www.lustre.org/ [ 479.086128] LNetError: 27587:0:(lib-move.c:2158:lnet_handle_find_routed_path()) peer 192.168.206.1@tcp99 has no available nets [ 481.297769] Key type lgssc unregistered [ 481.556975] LNetError: 28693:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 481.588914] LNet: Removed LNI 192.168.206.1@tcp [ 482.049831] Key type .llcrypt unregistered [ 482.051802] Key type ._llcrypt unregistered [ 486.911185] Key type ._llcrypt registered [ 486.914677] Key type .llcrypt registered [ 488.119962] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 00:23:37 (1788582217) [ 488.767272] Key type .llcrypt unregistered [ 488.768806] Key type ._llcrypt unregistered [ 494.617456] Key type ._llcrypt registered [ 494.619648] Key type .llcrypt registered [ 494.770675] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 494.784244] alg: No test for adler32 (adler32-zlib) [ 495.686374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 495.765770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 495.794873] LNet: 29602:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=16 less than recommended minimum 512 [ 495.812923] LNet: 29602:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 [ 495.837151] LNet: 29602:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 [ 495.857485] LNet: Message forwarding will be enabled in 2s [ 498.912682] LNet: Message forwarding enabled [ 499.327815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 499.350225] LNet: Message forwarding will stop in 2s to 6s [ 503.012458] LNet: Message forwarding disabled [ 503.845729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 503.878320] LNet: 29701:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=16 less than recommended minimum 512 [ 503.892296] LNet: 29701:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 [ 503.901389] LNet: 29701:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 [ 503.921542] LNet: Message forwarding will be enabled in 2s [ 504.057464] LNet: 29714:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 504.556411] Key type .llcrypt unregistered [ 504.558243] Key type ._llcrypt unregistered [ 514.431520] Key type ._llcrypt registered [ 514.434752] Key type .llcrypt registered [ 515.799410] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 00:24:04 (1788582244) [ 516.364524] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 516.398293] alg: No test for adler32 (adler32-zlib) [ 517.314825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 517.434583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 523.426565] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 00:24:12 (1788582252) [ 523.588314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 523.611554] LNet: 31112:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 523.627135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 523.676896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 523.703399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 523.731615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 523.752385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 523.774370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 523.800308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 523.832344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 523.866295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 523.917719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 523.996303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 524.040916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 524.083577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 524.142749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 524.193552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 524.269356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 524.316502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 524.359908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 524.431723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 524.480142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 524.505873] LNetError: 31155:0:(router.c:625:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 531.952367] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 00:24:20 (1788582260) [ 532.079466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 532.157904] LNet: 31720:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 532.212991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 532.356204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 539.865217] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 00:24:28 (1788582268) [ 541.917675] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg601-server [ 544.121541] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 00:24:32 (1788582272) [ 544.249242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 544.282459] LNet: 32653:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 544.298195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 544.419324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 544.460859] LNet: Added LNI 192.168.206.1@tcp [8/256/0/180] [ 545.340782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.206.[2-6]@tcp [ 545.362746] LNet: 32791:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 545.426401] LNetError: 32661:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.206.2@tcp is being used as a gateway but routing feature is not turned on [ 545.446914] LNetError: 32661:0:(lib-lnet.h:1280:lnet_set_route_aliveness()) route to tcp1 through 192.168.206.2@tcp has gone from up to down [ 551.227753] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 00:24:40 (1788582280) [ 552.962994] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg601-server [ 554.526266] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 00:24:43 (1788582283) [ 554.715349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 554.749258] LNet: 33725:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 554.755357] LNetError: 33725:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 554.770436] LNet: Removed LNI 192.168.206.1@tcp [ 554.791801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 554.865942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 554.892705] LNet: Added LNI 192.168.206.1@tcp [8/256/0/180] [ 555.462734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.206.[2-7/2]@tcp [ 555.501358] LNet: 33843:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 555.507949] LNet: 33843:0:(router.c:729:lnet_add_route()) Skipped 4 previous similar messages [ 555.535523] LNetError: 33730:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.206.2@tcp is being used as a gateway but routing feature is not turned on [ 555.539705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.206.[2-7/2]@tcp [ 555.548796] LNetError: 33730:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 555.548823] LNetError: 33730:0:(lib-lnet.h:1280:lnet_set_route_aliveness()) route to tcp103 through 192.168.206.2@tcp has gone from up to down [ 555.548827] LNetError: 33730:0:(lib-lnet.h:1280:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 560.551916] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 00:24:49 (1788582289) [ 560.626667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 560.641221] LNet: 34429:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 560.647319] LNetError: 34429:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 560.663316] LNet: Removed LNI 192.168.206.1@tcp [ 560.691932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 560.761847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 560.810650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 560.841894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 560.886652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 560.934233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 560.964286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 561.014896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 561.098869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 561.148140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 561.196285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 561.264528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 561.292927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 561.341746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 561.407732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 561.436340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 561.480934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 561.539460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 561.561395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 561.604078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 561.672795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 561.702215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 561.745535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 561.799758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 561.832722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 561.871511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 561.936443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 561.981853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 562.044895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 562.091853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 562.119355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 562.156464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 562.218324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 562.246558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 562.295603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 567.227055] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 00:24:56 (1788582296) [ 568.699898] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg601-server [ 569.993478] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 00:24:59 (1788582299) [ 571.485860] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg601-server [ 572.923680] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 00:25:02 (1788582302) [ 573.021130] LNet: 35828:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 573.032564] LNet: 35828:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 11 previous similar messages [ 573.666779] Key type .llcrypt unregistered [ 573.670280] Key type ._llcrypt unregistered [ 578.446703] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 578.595084] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 578.619196] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 578.624572] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 581.124307] Key type ._llcrypt registered [ 581.127241] Key type .llcrypt registered [ 581.276789] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 581.287130] alg: No test for adler32 (adler32-zlib) [ 582.194752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 582.264473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 582.282459] LNet: Added LNI 192.168.206.1@tcp [8/256/0/180] [ 582.793032] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 582.807124] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 584.199459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 584.222320] LNet: Removed LNI 10.1.2.3@tcp [ 584.300959] LNet: 36548:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 584.306662] LNetError: 36548:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 584.319931] LNet: Removed LNI 192.168.206.1@tcp [ 584.739266] Key type .llcrypt unregistered [ 584.741761] Key type ._llcrypt unregistered [ 595.508862] Key type ._llcrypt registered [ 595.511857] Key type .llcrypt registered [ 596.933810] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 00:25:25 (1788582325) [ 597.593517] Key type .llcrypt unregistered [ 597.596230] Key type ._llcrypt unregistered [ 604.038064] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 604.237766] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 604.246836] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 604.256408] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 606.918631] Key type ._llcrypt registered [ 606.920700] Key type .llcrypt registered [ 607.089701] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 607.143934] alg: No test for adler32 (adler32-zlib) [ 608.083869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 608.171775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 608.200965] LNet: Added LNI 192.168.206.1@tcp [8/256/0/180] [ 608.888187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 608.913652] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 611.237540] LNet: 38037:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 611.246022] LNetError: 38037:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 611.251813] LNet: Removed LNI 10.1.2.3@tcp [ 611.276864] LNet: Removed LNI 192.168.206.1@tcp [ 611.880987] Key type .llcrypt unregistered [ 611.883870] Key type ._llcrypt unregistered [ 621.843749] Key type ._llcrypt registered [ 621.845483] Key type .llcrypt registered [ 622.919412] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 00:25:51 (1788582351) [ 623.510437] Key type .llcrypt unregistered [ 623.512253] Key type ._llcrypt unregistered [ 629.517994] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 629.625864] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 629.640405] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 629.644509] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 632.393580] Key type ._llcrypt registered [ 632.395802] Key type .llcrypt registered [ 632.619190] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 632.629728] alg: No test for adler32 (adler32-zlib) [ 633.577708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 633.651409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 633.692029] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 635.634848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 635.654429] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 637.609686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 637.661699] LNet: Removed LNI 10.1.2.3@tcp [ 637.672729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 637.721973] LNet: Removed LNI 10.1.2.31@tcp [ 637.745229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 637.809297] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 638.849373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 638.871880] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 639.869398] LNet: 39954:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 639.877745] LNetError: 39954:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg:0 not found [ 639.888154] LNet: Removed LNI 10.1.2.31@tcp [ 639.899745] LNet: Removed LNI 10.1.2.3@tcp [ 640.601939] Key type .llcrypt unregistered [ 640.604519] Key type ._llcrypt unregistered [ 653.638794] Key type ._llcrypt registered [ 653.644199] Key type .llcrypt registered [ 654.887406] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 00:26:23 (1788582383) [ 655.299642] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 655.310396] alg: No test for adler32 (adler32-zlib) [ 656.181486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 656.227948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 656.252100] LNet: Added LNI 192.168.206.1@tcp [8/256/0/180] [ 657.024887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 657.050532] LNet: 40906:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 657.066595] LNetError: 40906:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 657.092842] LNet: Removed LNI 192.168.206.1@tcp [ 657.112543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 657.192632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.206.1@tcp [ 657.236137] LNet: Added LNI 192.168.206.1@tcp [8/256/0/180] [ 657.951302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.206.1@tcp [ 657.976645] LNet: Removed LNI 192.168.206.1@tcp [ 658.014588] LNet: 41041:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 658.609408] Key type .llcrypt unregistered [ 658.613818] Key type ._llcrypt unregistered [ 668.140788] Key type ._llcrypt registered [ 668.142499] Key type .llcrypt registered [ 669.336920] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 00:26:38 (1788582398) [ 670.825550] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg601-server [ 672.174239] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 00:26:41 (1788582401) [ 672.653287] Key type .llcrypt unregistered [ 672.657472] Key type ._llcrypt unregistered [ 677.921747] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 678.112876] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 678.127454] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 678.134781] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 680.552802] Key type ._llcrypt registered [ 680.555105] Key type .llcrypt registered [ 680.682269] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 680.694682] alg: No test for adler32 (adler32-zlib) [ 681.612970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 681.659514] LNet: Added LNI 192.168.206.1@tcp1 [8/256/0/180] [ 682.127903] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 683.818836] LNet: 42872:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 683.828233] LNetError: 42872:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 683.840747] LNet: Removed LNI 192.168.206.1@tcp1 [ 683.864159] LNet: Removed LNI 10.1.2.3@tcp2 [ 684.293682] Key type .llcrypt unregistered [ 684.297090] Key type ._llcrypt unregistered [ 693.325899] Key type ._llcrypt registered [ 693.329603] Key type .llcrypt registered [ 694.333114] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 00:27:03 (1788582423) [ 694.704275] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 694.728681] alg: No test for adler32 (adler32-zlib) [ 695.587188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 695.630378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 695.647222] LNet: Added LNI 192.168.206.1@tcp [8/256/0/180] [ 696.262674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 696.276188] LNet: 43839:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 696.284502] LNetError: 43839:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 696.295210] LNet: Removed LNI 192.168.206.1@tcp [ 696.319294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 696.366497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 696.386897] LNet: Added LNI 192.168.206.1@tcp [8/256/0/180] [ 701.611831] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 00:27:10 (1788582430) [ 701.763242] LNet: 44424:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 701.774337] LNetError: 44424:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 701.811969] LNet: Removed LNI 192.168.206.1@tcp [ 702.392576] Key type .llcrypt unregistered [ 702.398531] Key type ._llcrypt unregistered [ 707.081271] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 707.262269] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 707.270156] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 707.279833] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 709.747717] Key type ._llcrypt registered [ 709.751566] Key type .llcrypt registered [ 709.969130] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 709.987613] alg: No test for adler32 (adler32-zlib) [ 710.934330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 710.978158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 711.004288] LNet: Added LNI 192.168.206.1@tcp [8/256/0/180] [ 711.660879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 711.695701] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 713.463164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 713.483141] LNet: 45169:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 713.489280] LNetError: 45169:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 713.495948] LNet: Removed LNI 10.1.2.3@tcp [ 713.509889] LNet: Removed LNI 192.168.206.1@tcp [ 713.540871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 713.620988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 713.655932] LNet: Added LNI 192.168.206.1@tcp [8/256/0/180] [ 713.668025] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 714.216659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 714.240389] LNet: 45197:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 714.251963] LNetError: 45197:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 714.260445] LNetError: 45197:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 714.271395] LNet: Removed LNI 10.1.2.3@tcp [ 714.290868] LNet: Removed LNI 192.168.206.1@tcp [ 714.329752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 714.400723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 714.421440] LNet: Added LNI 192.168.206.1@tcp [8/256/0/180] [ 714.425856] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 714.944898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 714.970668] LNet: Added LNI 192.168.206.1@tcp2 [8/256/0/180] [ 715.880772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 715.903761] LNet: 45359:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 715.910238] LNetError: 45359:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 715.916903] LNetError: 45359:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 715.925471] LNet: Removed LNI 10.1.2.3@tcp [ 715.930522] LNet: Removed LNI 192.168.206.1@tcp [ 715.941765] LNet: Removed LNI 192.168.206.1@tcp2 [ 715.963532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 716.018881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 716.042519] LNet: Added LNI 192.168.206.1@tcp [8/256/0/180] [ 716.046852] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 716.054001] LNet: Added LNI 192.168.206.1@tcp2 [8/256/0/180] [ 722.031658] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 00:27:31 (1788582451) [ 722.236570] LNet: 45956:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 722.241692] LNetError: 45956:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 722.248789] LNetError: 45956:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 722.253397] LNet: Removed LNI 10.1.2.3@tcp [ 722.260432] LNet: Removed LNI 192.168.206.1@tcp [ 722.269989] LNet: Removed LNI 192.168.206.1@tcp2 [ 722.742181] Key type .llcrypt unregistered [ 722.745637] Key type ._llcrypt unregistered [ 727.746825] Key type ._llcrypt registered [ 727.751023] Key type .llcrypt registered [ 727.894931] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 727.908951] alg: No test for adler32 (adler32-zlib) [ 728.777968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 728.823341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 728.845797] LNet: Added LNI 192.168.206.1@tcp [8/256/0/180] [ 729.709389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 729.730334] LNet: 46346:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 729.734990] LNetError: 46346:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 729.750053] LNet: Removed LNI 192.168.206.1@tcp [ 729.782203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 729.849020] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 729.883541] LNet: Added LNI 192.168.206.1@tcp [8/256/0/180] [ 730.323360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 730.342821] LNet: 46376:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 730.349208] LNetError: 46376:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 730.361448] LNet: Removed LNI 192.168.206.1@tcp [ 730.380991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 730.429737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 730.451553] LNet: Added LNI 192.168.206.1@tcp [8/256/0/180] [ 730.957480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 730.990056] LNet: Removed LNI 192.168.206.1@tcp [ 731.012827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 731.073482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 731.117253] LNet: Added LNI 192.168.206.1@tcp [8/256/0/180] [ 731.559181] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 731.707140] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 731.711596] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 731.717720] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 734.034586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 734.054383] LNet: 46484:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 734.059459] LNet: 46484:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 734.066675] LNetError: 46484:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 734.070875] LNetError: 46484:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 734.085949] LNet: Removed LNI 192.168.206.1@tcp [ 734.108762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 734.172879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 734.191744] LNet: Added LNI 192.168.206.1@tcp [8/256/0/180] [ 739.070993] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 00:27:48 (1788582468) [ 739.280285] LNet: 47082:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 739.294690] LNetError: 47082:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 739.311924] LNet: Removed LNI 192.168.206.1@tcp [ 739.758731] Key type .llcrypt unregistered [ 739.760959] Key type ._llcrypt unregistered [ 743.790391] Key type ._llcrypt registered [ 743.793025] Key type .llcrypt registered [ 743.968972] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 743.982523] alg: No test for adler32 (adler32-zlib) [ 744.871638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 744.922662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 744.943330] LNet: Added LNI 192.168.206.1@tcp [8/256/0/180] [ 745.612631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 745.628784] LNet: 47477:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 745.633913] LNetError: 47477:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 745.649329] LNet: Removed LNI 192.168.206.1@tcp [ 745.677296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 745.738815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 745.758929] LNet: Added LNI 192.168.206.1@tcp [8/256/0/180] [ 750.879901] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 00:28:00 (1788582480) [ 751.018480] LNet: 48066:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 751.030356] LNetError: 48066:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 751.060748] LNet: Removed LNI 192.168.206.1@tcp [ 751.583442] Key type .llcrypt unregistered [ 751.586108] Key type ._llcrypt unregistered [ 756.071360] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 756.153592] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 756.156975] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 756.166206] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 758.554796] Key type ._llcrypt registered [ 758.556612] Key type .llcrypt registered [ 758.690150] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 758.704429] alg: No test for adler32 (adler32-zlib) [ 759.573778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 759.616795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 759.636921] LNet: Added LNI 192.168.206.1@tcp [8/256/0/180] [ 760.316067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 760.336841] LNet: 48513:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 760.342818] LNetError: 48513:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 760.358436] LNet: Removed LNI 192.168.206.1@tcp [ 760.374816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 760.423974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 760.449534] LNet: Added LNI 192.168.206.1@tcp [8/256/0/180] [ 765.528448] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 00:28:14 (1788582494) [ 765.621034] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 765.654271] LNet: 49105:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 765.660625] LNetError: 49105:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 765.677817] LNet: Removed LNI 192.168.206.1@tcp [ 765.721438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 765.806682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 770.668953] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 00:28:19 (1788582499) [ 770.779502] LNet: 49690:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 771.244834] Key type .llcrypt unregistered [ 771.247478] Key type ._llcrypt unregistered [ 775.051563] Key type ._llcrypt registered [ 775.053242] Key type .llcrypt registered [ 775.184886] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 775.195849] alg: No test for adler32 (adler32-zlib) [ 776.036624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 776.078658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 776.095591] LNet: Added LNI 192.168.206.1@tcp [8/256/0/180] [ 776.702347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 776.731317] LNet: 50092:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 776.740628] LNetError: 50092:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 776.753167] LNet: Removed LNI 192.168.206.1@tcp [ 776.770617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 776.823505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 776.843057] LNet: Added LNI 192.168.206.1@tcp [8/256/0/180] [ 781.780532] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 00:28:30 (1788582510) [ 781.846544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 781.870568] LNet: 50677:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 781.877535] LNetError: 50677:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 781.895533] LNet: Removed LNI 192.168.206.1@tcp [ 781.909666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 781.985202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 786.891433] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 00:28:36 (1788582516) [ 788.143612] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 789.341647] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 00:28:38 (1788582518) [ 790.688293] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 791.956374] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 00:28:41 (1788582521) [ 793.172688] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 794.472281] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 00:28:43 (1788582523) [ 795.646473] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 797.305642] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 00:28:46 (1788582526) [ 798.743054] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 800.353760] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing ========================================================== 00:28:49 (1788582529) [ 800.500134] LNet: 52729:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 801.060073] Key type .llcrypt unregistered [ 801.066801] Key type ._llcrypt unregistered [ 806.882628] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 807.060581] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 807.066660] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 807.074605] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 809.716730] Key type ._llcrypt registered [ 809.719475] Key type .llcrypt registered [ 810.056160] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 810.075429] alg: No test for adler32 (adler32-zlib) [ 810.975914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 811.047963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 811.071735] LNet: Added LNI 192.168.206.1@tcp [8/512/0/180] [ 811.079278] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 811.081608] LNet: 53047:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 811.158884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 811.187471] LNet: 53066:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 811.199417] LNetError: 53066:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 811.211195] LNet: Removed LNI 10.1.2.3@tcp [ 811.213137] LNetError: 53066:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 811.236287] LNet: Removed LNI 192.168.206.1@tcp [ 811.258572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 811.325663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 811.341244] LNet: Added LNI 192.168.206.1@tcp [8/256/0/180] [ 811.392595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 811.420735] LNet: 53090:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 811.427762] LNetError: 53090:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 811.443718] LNet: Removed LNI 192.168.206.1@tcp [ 811.460829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 811.521453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 811.560633] LNet: Added LNI 192.168.206.1@tcp [8/256/0/180] [ 811.570291] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 811.573510] LNet: 53098:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 811.664959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 811.686503] LNet: 53116:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 811.693778] LNetError: 53116:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 811.701176] LNet: Removed LNI 10.1.2.3@tcp [ 811.703190] LNetError: 53116:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 811.712686] LNet: Removed LNI 192.168.206.1@tcp [ 811.725900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 811.771134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 811.795176] LNet: Added LNI 192.168.206.1@tcp [8/256/0/180] [ 811.800938] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 811.807213] LNet: 53124:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 811.999293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 812.021316] LNet: 53150:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 812.029427] LNetError: 53150:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 812.036854] LNet: Removed LNI 10.1.2.3@tcp [ 812.047385] LNetError: 53150:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 812.062764] LNet: Removed LNI 192.168.206.1@tcp [ 812.092994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 812.152776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 812.172273] LNet: Added LNI 192.168.206.1@tcp [8/256/0/182] [ 812.175965] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 812.262563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 812.291407] LNet: 53178:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 812.298443] LNetError: 53178:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 812.304657] LNet: Removed LNI 10.1.2.3@tcp [ 812.307341] LNetError: 53178:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 812.318244] LNet: Removed LNI 192.168.206.1@tcp [ 812.331735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 812.388398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 812.404426] LNet: Added LNI 192.168.206.1@tcp [8/256/0/182] [ 812.408368] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 812.410268] LNet: 53186:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 812.470589] LNet: Removed LNI 10.1.2.3@tcp [ 812.484358] LNet: Removed LNI 192.168.206.1@tcp [ 813.020630] Key type .llcrypt unregistered [ 813.022589] Key type ._llcrypt unregistered [ 822.410496] Key type ._llcrypt registered [ 822.413933] Key type .llcrypt registered [ 823.556300] Lustre: DEBUG MARKER: == sanity-lnet test 180: FastReg map error should not leak descriptors ========================================================== 00:29:12 (1788582552) [ 824.661101] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_180 Need o2ib NETTYPE [ 826.211516] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 00:29:15 (1788582555) [ 826.440378] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 826.452972] alg: No test for adler32 (adler32-zlib) [ 827.411110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 827.474956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 827.529498] LNet: Added LNI 192.168.206.1@tcp [8/256/0/180] [ 828.352640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.206.1 [ 828.379129] LNet: 54447:0:(config.c:1738:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 828.397672] LNet: Removed LNI 192.168.206.1@tcp [ 833.829348] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 00:29:22 (1788582562) [ 834.038067] LNet: 55017:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 834.045992] LNetError: 55017:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 834.075869] LNet: Removed LNI 192.168.206.1@tcp [ 834.642359] Key type .llcrypt unregistered [ 834.647233] Key type ._llcrypt unregistered [ 840.377161] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 840.584728] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 843.386579] Key type ._llcrypt registered [ 843.391794] Key type .llcrypt registered [ 843.622399] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 843.648858] alg: No test for adler32 (adler32-zlib) [ 845.061708] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 851.320726] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 00:29:40 (1788582580) [ 851.394515] LNet: 55918:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 851.401528] LNetError: 55918:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 851.425309] LNet: Removed LNI 10.1.2.3@tcp [ 852.065919] Key type .llcrypt unregistered [ 852.067986] Key type ._llcrypt unregistered [ 858.074059] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 858.341307] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 858.348514] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 858.354582] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 861.053374] Key type ._llcrypt registered [ 861.055724] Key type .llcrypt registered [ 861.179457] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 861.192690] alg: No test for adler32 (adler32-zlib) [ 862.210990] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 868.820661] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 00:29:57 (1788582597) [ 871.026130] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 873.201445] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 00:30:01 (1788582601) [ 873.324722] LNet: 57103:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 873.335161] LNetError: 57103:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 873.372156] LNet: Removed LNI 10.1.2.3@tcp [ 874.098676] Key type .llcrypt unregistered [ 874.102062] Key type ._llcrypt unregistered [ 879.811370] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 880.067170] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 880.069836] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 880.075292] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 881.630194] Key type ._llcrypt registered [ 881.633427] Key type .llcrypt registered [ 881.830396] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 881.846293] alg: No test for adler32 (adler32-zlib) [ 882.803767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 882.852871] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 891.802531] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 00:30:20 (1788582620) [ 892.421346] LNet: 58325:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 892.429504] LNetError: 58325:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 892.462037] LNet: Removed LNI 10.1.2.3@tcp [ 893.177324] Key type .llcrypt unregistered [ 893.179149] Key type ._llcrypt unregistered [ 899.035254] Key type ._llcrypt registered [ 899.041890] Key type .llcrypt registered [ 899.238919] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 899.253979] alg: No test for adler32 (adler32-zlib) [ 900.173798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 900.211072] LNet: Added LNI 192.168.206.1@tcp [8/256/0/180] [ 908.413139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.101@tcp [ 909.685455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.1@tcp -d 192.168.206.101@tcp -m GET -e local_interrupt -r 1 [ 909.755090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.1@tcp -d 192.168.206.1@tcp -m GET -e local_interrupt -r 1 [ 909.791995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.101@tcp [ 909.816756] LNet: There was an unexpected network error while writing to 192.168.206.101: rc = -22 [ 910.123818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 910.160884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 910.187588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 910.212421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 910.234982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 910.264987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 910.287710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 910.322441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 910.339904] LNet: 58760:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 910.355389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 910.393960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 910.428699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 910.459773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 910.498297] LNet: 1 local NIs in recovery (showing 1): 192.168.206.1@tcp [ 910.503846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 910.504785] LNet: There was an unexpected network error while writing to 192.168.206.1: rc = -22 [ 910.550551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 910.620352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 911.264201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.1@tcp -d 192.168.206.101@tcp -m GET -e local_dropped -r 1 [ 911.317962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.1@tcp -d 192.168.206.1@tcp -m GET -e local_dropped -r 1 [ 911.355572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.101@tcp [ 911.629600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 911.663683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 911.704848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 911.740697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 911.779770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 911.822670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 911.884730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 911.933811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 911.950517] LNet: 58869:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 911.967366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 911.990733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 912.027267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 912.083987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 912.132905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 912.169483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 912.204799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 912.798292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.1@tcp -d 192.168.206.101@tcp -m GET -e local_aborted -r 1 [ 912.855989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.1@tcp -d 192.168.206.1@tcp -m GET -e local_aborted -r 1 [ 912.890866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.101@tcp [ 912.907913] LNet: There was an unexpected network error while writing to 192.168.206.101: rc = -22 [ 912.911652] LNet: Skipped 1 previous similar message [ 913.184798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 913.219929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 913.293506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 913.345780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 913.400917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 913.448828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 913.517436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 913.545409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 913.559768] LNet: 58978:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 913.583790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 913.620368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 913.658346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 913.704906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 913.754402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 913.779568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 913.809290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 914.406243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.1@tcp -d 192.168.206.101@tcp -m GET -e local_no_route -r 1 [ 914.438964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.1@tcp -d 192.168.206.1@tcp -m GET -e local_no_route -r 1 [ 914.465324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.101@tcp [ 914.683658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 914.717281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 914.741109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 914.795727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 914.840067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 914.871957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 914.900771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 914.924405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 914.965861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 915.008946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 915.036476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 915.070501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 915.115681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 915.187425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 915.220375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 915.704678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.1@tcp -d 192.168.206.101@tcp -m GET -e local_timeout -r 1 [ 915.738636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.1@tcp -d 192.168.206.1@tcp -m GET -e local_timeout -r 1 [ 915.770577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.101@tcp [ 915.790597] LNet: There was an unexpected network error while writing to 192.168.206.101: rc = -22 [ 915.795437] LNet: Skipped 2 previous similar messages [ 916.030554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 916.064457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 916.092404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 916.120158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 916.155546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 916.202675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 916.251628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 916.307049] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 916.336681] LNet: 59197:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 916.341349] LNet: 59197:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 916.357217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 916.387547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 916.414440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 916.438305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 916.464275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 916.490697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 916.521643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 917.039993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.1@tcp -d 192.168.206.101@tcp -m GET -e local_error -r 1 [ 917.086901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.1@tcp -d 192.168.206.1@tcp -m GET -e local_error -r 1 [ 917.128792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.101@tcp [ 917.420770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 917.451053] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 917.486273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 917.541744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 917.592953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 917.666466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 917.720014] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 917.780853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 917.828670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 917.869210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 917.910436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 917.947508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 917.992212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 918.051121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 918.110323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 918.414875] LNet: 59347:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 918.422401] LNetError: 59347:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 918.458051] LNet: Removed LNI 192.168.206.1@tcp [ 918.939896] Key type .llcrypt unregistered [ 918.941832] Key type ._llcrypt unregistered [ 935.614946] Key type ._llcrypt registered [ 935.618126] Key type .llcrypt registered [ 937.238580] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 00:31:05 (1788582665) [ 938.409080] Key type .llcrypt unregistered [ 938.411646] Key type ._llcrypt unregistered [ 945.654555] Key type ._llcrypt registered [ 945.656886] Key type .llcrypt registered [ 945.849792] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 945.858871] alg: No test for adler32 (adler32-zlib) [ 946.741477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 946.761249] LNet: Added LNI 192.168.206.1@tcp [8/256/0/180] [ 954.950874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.101@tcp [ 965.889696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 965.911978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 965.923452] LNet: Added LNI 192.168.206.1@tcp1 [8/256/0/180] [ 967.556835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.1@tcp -d 192.168.206.101@tcp -m GET -e local_interrupt -r 1 [ 967.600311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.1@tcp -d 192.168.206.101@tcp1 -m GET -e local_interrupt -r 1 [ 967.634990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.1@tcp -d 192.168.206.1@tcp -m GET -e local_interrupt -r 1 [ 967.664167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.1@tcp -d 192.168.206.1@tcp1 -m GET -e local_interrupt -r 1 [ 967.700595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.1@tcp1 -d 192.168.206.101@tcp -m GET -e local_interrupt -r 1 [ 967.737256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.1@tcp1 -d 192.168.206.101@tcp1 -m GET -e local_interrupt -r 1 [ 967.776814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.1@tcp1 -d 192.168.206.1@tcp -m GET -e local_interrupt -r 1 [ 967.817699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.1@tcp1 -d 192.168.206.1@tcp1 -m GET -e local_interrupt -r 1 [ 967.857567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.101@tcp [ 967.878466] LNet: There was an unexpected network error while writing to 192.168.206.101: rc = -22 [ 967.886599] LNet: 1 local NIs in recovery (showing 1): 192.168.206.1@tcp [ 968.089935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 968.112747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 968.138087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 968.164434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 968.193876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 968.219962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 968.244318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 968.276128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 968.305294] LNet: 60811:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 968.330597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 968.360971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 968.383647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 968.403905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 968.430915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 968.463894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 968.497229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 968.914565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.1@tcp -d 192.168.206.101@tcp -m GET -e local_dropped -r 1 [ 968.938989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.1@tcp -d 192.168.206.101@tcp1 -m GET -e local_dropped -r 1 [ 968.969848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.1@tcp -d 192.168.206.1@tcp -m GET -e local_dropped -r 1 [ 969.013342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.1@tcp -d 192.168.206.1@tcp1 -m GET -e local_dropped -r 1 [ 969.048866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.1@tcp1 -d 192.168.206.101@tcp -m GET -e local_dropped -r 1 [ 969.084714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.1@tcp1 -d 192.168.206.101@tcp1 -m GET -e local_dropped -r 1 [ 969.131889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.1@tcp1 -d 192.168.206.1@tcp -m GET -e local_dropped -r 1 [ 969.184418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.1@tcp1 -d 192.168.206.1@tcp1 -m GET -e local_dropped -r 1 [ 969.225583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.101@tcp [ 969.240555] LNet: There was an unexpected network error while writing to 192.168.206.101: rc = -22 [ 969.244363] LNet: Skipped 2 previous similar messages [ 969.444892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 969.469986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 969.499189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 969.531497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 969.561964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 969.590829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 969.613874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 969.640483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 969.654969] LNet: 60932:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 969.671537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 969.696559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 969.717459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 969.738818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 969.762781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 969.792178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 969.822332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 970.348689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.1@tcp -d 192.168.206.101@tcp -m GET -e local_aborted -r 1 [ 970.396527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.1@tcp -d 192.168.206.101@tcp1 -m GET -e local_aborted -r 1 [ 970.435994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.1@tcp -d 192.168.206.1@tcp -m GET -e local_aborted -r 1 [ 970.481404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.1@tcp -d 192.168.206.1@tcp1 -m GET -e local_aborted -r 1 [ 970.533936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.1@tcp1 -d 192.168.206.101@tcp -m GET -e local_aborted -r 1 [ 970.575861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.1@tcp1 -d 192.168.206.101@tcp1 -m GET -e local_aborted -r 1 [ 970.620788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.1@tcp1 -d 192.168.206.1@tcp -m GET -e local_aborted -r 1 [ 970.669572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.1@tcp1 -d 192.168.206.1@tcp1 -m GET -e local_aborted -r 1 [ 970.725068] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.101@tcp [ 970.747561] LNet: There was an unexpected network error while writing to 192.168.206.101: rc = -22 [ 970.754825] LNet: Skipped 2 previous similar messages [ 971.044932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 971.095815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 971.136060] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 971.188480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 971.242576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 971.279340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 971.324735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 971.363715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 971.390223] LNet: 61054:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 971.414410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 971.447388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 971.474882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 971.517553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 971.546479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 971.576138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 971.607839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 972.011695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.1@tcp -d 192.168.206.101@tcp -m GET -e local_no_route -r 1 [ 972.041865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.1@tcp -d 192.168.206.101@tcp1 -m GET -e local_no_route -r 1 [ 972.075516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.1@tcp -d 192.168.206.1@tcp -m GET -e local_no_route -r 1 [ 972.110560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.1@tcp -d 192.168.206.1@tcp1 -m GET -e local_no_route -r 1 [ 972.152322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.1@tcp1 -d 192.168.206.101@tcp -m GET -e local_no_route -r 1 [ 972.205130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.1@tcp1 -d 192.168.206.101@tcp1 -m GET -e local_no_route -r 1 [ 972.243407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.1@tcp1 -d 192.168.206.1@tcp -m GET -e local_no_route -r 1 [ 972.279938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.1@tcp1 -d 192.168.206.1@tcp1 -m GET -e local_no_route -r 1 [ 972.325426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.101@tcp [ 972.574281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 972.607586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 972.655823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 972.701371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 972.738836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 972.784870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 972.817834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 972.844867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 972.880542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 972.914609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 972.943841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 972.973777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 973.011377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 973.038693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 973.076954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 973.625645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.1@tcp -d 192.168.206.101@tcp -m GET -e local_timeout -r 1 [ 973.688650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.1@tcp -d 192.168.206.101@tcp1 -m GET -e local_timeout -r 1 [ 973.731446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.1@tcp -d 192.168.206.1@tcp -m GET -e local_timeout -r 1 [ 973.767827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.1@tcp -d 192.168.206.1@tcp1 -m GET -e local_timeout -r 1 [ 973.805761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.1@tcp1 -d 192.168.206.101@tcp -m GET -e local_timeout -r 1 [ 973.835593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.1@tcp1 -d 192.168.206.101@tcp1 -m GET -e local_timeout -r 1 [ 973.888286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.1@tcp1 -d 192.168.206.1@tcp -m GET -e local_timeout -r 1 [ 973.937651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.1@tcp1 -d 192.168.206.1@tcp1 -m GET -e local_timeout -r 1 [ 973.983434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.101@tcp [ 974.006972] LNet: There was an unexpected network error while writing to 192.168.206.101: rc = -22 [ 974.013151] LNet: Skipped 5 previous similar messages [ 974.205398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 974.226638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 974.261798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 974.287992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 974.323423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 974.360925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 974.405511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 974.448507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 974.463711] LNet: 61296:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 974.469803] LNet: 61296:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 974.485458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 974.514423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 974.553195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 974.589381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 974.617729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 974.653568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 974.693354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 975.181746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.1@tcp -d 192.168.206.101@tcp -m GET -e local_error -r 1 [ 975.223782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.1@tcp -d 192.168.206.101@tcp1 -m GET -e local_error -r 1 [ 975.256963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.1@tcp -d 192.168.206.1@tcp -m GET -e local_error -r 1 [ 975.295575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.1@tcp -d 192.168.206.1@tcp1 -m GET -e local_error -r 1 [ 975.337342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.1@tcp1 -d 192.168.206.101@tcp -m GET -e local_error -r 1 [ 975.371812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.1@tcp1 -d 192.168.206.101@tcp1 -m GET -e local_error -r 1 [ 975.399485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.1@tcp1 -d 192.168.206.1@tcp -m GET -e local_error -r 1 [ 975.435156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.1@tcp1 -d 192.168.206.1@tcp1 -m GET -e local_error -r 1 [ 975.465898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.101@tcp [ 975.648913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 975.697383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 975.717962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 975.740518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 975.765639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 975.790442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 975.816096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 975.839288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 975.867551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 975.903446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 975.928938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 975.975226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 976.018983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 976.053994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 976.097533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 981.453209] LNet: 61493:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 981.465601] LNetError: 61493:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 981.474361] LNet: Removed LNI 192.168.206.1@tcp [ 981.484875] LNet: Removed LNI 192.168.206.1@tcp1 [ 981.789330] Key type .llcrypt unregistered [ 981.791705] Key type ._llcrypt unregistered [ 996.489596] Key type ._llcrypt registered [ 996.492876] Key type .llcrypt registered [ 997.448965] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 00:32:06 (1788582726) [ 998.103638] Key type .llcrypt unregistered [ 998.105794] Key type ._llcrypt unregistered [ 1002.900710] Key type ._llcrypt registered [ 1002.902283] Key type .llcrypt registered [ 1003.097934] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1003.114764] alg: No test for adler32 (adler32-zlib) [ 1004.079257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1004.126085] LNet: Added LNI 192.168.206.1@tcp [8/256/0/180] [ 1011.474706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.101@tcp [ 1012.115509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.1@tcp -d 192.168.206.101@tcp -m GET -e remote_dropped -r 1 [ 1012.161947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.1@tcp -d 192.168.206.1@tcp -m GET -e remote_dropped -r 1 [ 1012.217440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.101@tcp [ 1012.240840] LNet: There was an unexpected network error while writing to 192.168.206.101: rc = -22 [ 1012.435301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1012.461201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1012.483965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1012.505451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1012.536641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1012.572971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1012.604477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1012.632684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1012.652524] LNet: 62737:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1012.674960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1012.726537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1012.772577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1012.816192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1012.853262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1012.893708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1012.938927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1013.516619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.1@tcp -d 192.168.206.101@tcp -m GET -e remote_error -r 1 [ 1013.545314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.1@tcp -d 192.168.206.1@tcp -m GET -e remote_error -r 1 [ 1013.575318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.101@tcp [ 1013.592687] LNet: There was an unexpected network error while writing to 192.168.206.101: rc = -22 [ 1013.807818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1013.851594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1013.873367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1013.896564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1013.941430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1013.965033] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1014.003585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1014.039204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1014.070889] LNet: 62846:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1014.106750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1014.154147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1014.201734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1014.277580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1014.333258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1014.435668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1014.502920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1015.026901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.1@tcp -d 192.168.206.101@tcp -m GET -e remote_timeout -r 1 [ 1015.057730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.1@tcp -d 192.168.206.1@tcp -m GET -e remote_timeout -r 1 [ 1015.082571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.101@tcp [ 1015.101457] LNet: There was an unexpected network error while writing to 192.168.206.101: rc = -22 [ 1015.392272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1015.444702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1015.498166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1015.546741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1015.582851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1015.631954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1015.669507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1015.694447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1015.709422] LNet: 62955:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1015.727681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1015.751590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1015.777661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1015.803850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1015.835632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1015.863198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1015.910554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1016.208360] LNet: 62996:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1016.214308] LNetError: 62996:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1016.236308] LNet: Removed LNI 192.168.206.1@tcp [ 1016.792677] Key type .llcrypt unregistered [ 1016.796364] Key type ._llcrypt unregistered [ 1033.544567] Key type ._llcrypt registered [ 1033.547364] Key type .llcrypt registered [ 1034.680818] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 00:32:43 (1788582763) [ 1035.571296] Key type .llcrypt unregistered [ 1035.576450] Key type ._llcrypt unregistered [ 1041.947922] Key type ._llcrypt registered [ 1041.951281] Key type .llcrypt registered [ 1042.174699] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1042.190125] alg: No test for adler32 (adler32-zlib) [ 1043.194908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1043.222579] LNet: Added LNI 192.168.206.1@tcp [8/256/0/180] [ 1053.596485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.101@tcp [ 1063.874765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1063.910727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1063.956056] LNet: Added LNI 192.168.206.1@tcp1 [8/256/0/180] [ 1065.283800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.1@tcp -d 192.168.206.101@tcp -m GET -e remote_dropped -r 1 [ 1065.328471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.1@tcp -d 192.168.206.101@tcp1 -m GET -e remote_dropped -r 1 [ 1065.391581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.1@tcp -d 192.168.206.1@tcp -m GET -e remote_dropped -r 1 [ 1065.430172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.1@tcp -d 192.168.206.1@tcp1 -m GET -e remote_dropped -r 1 [ 1065.489432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.1@tcp1 -d 192.168.206.101@tcp -m GET -e remote_dropped -r 1 [ 1065.543583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.1@tcp1 -d 192.168.206.101@tcp1 -m GET -e remote_dropped -r 1 [ 1065.618133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.1@tcp1 -d 192.168.206.1@tcp -m GET -e remote_dropped -r 1 [ 1065.679470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.1@tcp1 -d 192.168.206.1@tcp1 -m GET -e remote_dropped -r 1 [ 1065.755918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.101@tcp [ 1065.782547] LNet: There was an unexpected network error while writing to 192.168.206.101: rc = -22 [ 1065.789505] LNet: 1 peer NIs in recovery (showing 1): 192.168.206.101@tcp [ 1066.054551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1066.094088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1066.166348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1066.219555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1066.268063] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1066.314953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1066.354283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1066.388966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1066.409160] LNet: 64458:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1066.420471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1066.449873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1066.477111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1066.499540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1066.521224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1066.540734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1066.593516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1066.621243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1067.177390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.1@tcp -d 192.168.206.101@tcp -m GET -e remote_error -r 1 [ 1067.207615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.1@tcp -d 192.168.206.101@tcp1 -m GET -e remote_error -r 1 [ 1067.237451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.1@tcp -d 192.168.206.1@tcp -m GET -e remote_error -r 1 [ 1067.262285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.1@tcp -d 192.168.206.1@tcp1 -m GET -e remote_error -r 1 [ 1067.322925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.1@tcp1 -d 192.168.206.101@tcp -m GET -e remote_error -r 1 [ 1067.402337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.1@tcp1 -d 192.168.206.101@tcp1 -m GET -e remote_error -r 1 [ 1067.455552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.1@tcp1 -d 192.168.206.1@tcp -m GET -e remote_error -r 1 [ 1067.512560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.1@tcp1 -d 192.168.206.1@tcp1 -m GET -e remote_error -r 1 [ 1067.556338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.101@tcp [ 1067.572955] LNet: There was an unexpected network error while writing to 192.168.206.101: rc = -22 [ 1067.577771] LNet: Skipped 2 previous similar messages [ 1067.773494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1067.807309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1067.838430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1067.867954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1067.917126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1067.956222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1067.999542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1068.035464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1068.067294] LNet: 64582:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1068.087117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1068.121424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1068.162253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1068.197541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1068.222899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1068.252504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1068.304531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1068.347840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1068.837865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.1@tcp -d 192.168.206.101@tcp -m GET -e remote_timeout -r 1 [ 1068.887338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.1@tcp -d 192.168.206.101@tcp1 -m GET -e remote_timeout -r 1 [ 1068.924284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.1@tcp -d 192.168.206.1@tcp -m GET -e remote_timeout -r 1 [ 1068.954361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.1@tcp -d 192.168.206.1@tcp1 -m GET -e remote_timeout -r 1 [ 1068.992611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.1@tcp1 -d 192.168.206.101@tcp -m GET -e remote_timeout -r 1 [ 1069.032640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.1@tcp1 -d 192.168.206.101@tcp1 -m GET -e remote_timeout -r 1 [ 1069.065869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.1@tcp1 -d 192.168.206.1@tcp -m GET -e remote_timeout -r 1 [ 1069.100955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.1@tcp1 -d 192.168.206.1@tcp1 -m GET -e remote_timeout -r 1 [ 1069.143562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.101@tcp [ 1069.163476] LNet: There was an unexpected network error while writing to 192.168.206.101: rc = -22 [ 1069.420891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1069.467779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1069.496548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1069.525310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1069.555625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1069.583279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1069.609565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1069.639590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1069.654272] LNet: 64705:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1069.670962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1069.705367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1069.769056] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1069.808725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1069.836805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1069.862534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1069.889573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1069.917537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1073.926896] LNet: 64782:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1073.932531] LNetError: 64782:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1073.942238] LNet: Removed LNI 192.168.206.1@tcp [ 1073.957727] LNet: Removed LNI 192.168.206.1@tcp1 [ 1074.302049] Key type .llcrypt unregistered [ 1074.305094] Key type ._llcrypt unregistered [ 1088.347765] Key type ._llcrypt registered [ 1088.354045] Key type .llcrypt registered [ 1089.366413] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 00:33:38 (1788582818) [ 1090.817884] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 1092.557896] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 00:33:41 (1788582821) [ 1093.424558] Key type .llcrypt unregistered [ 1093.432055] Key type ._llcrypt unregistered [ 1099.046203] Key type ._llcrypt registered [ 1099.048824] Key type .llcrypt registered [ 1099.255895] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1099.264893] alg: No test for adler32 (adler32-zlib) [ 1100.258215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1100.307416] LNet: Added LNI 192.168.206.1@tcp [8/256/0/180] [ 1107.306524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.101@tcp [ 1108.458472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.1@tcp -d 192.168.206.101@tcp -m GET -e network_timeout -r 1 [ 1108.490165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.1@tcp -d 192.168.206.1@tcp -m GET -e network_timeout -r 1 [ 1108.520267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.101@tcp [ 1108.540775] LNet: There was an unexpected network error while writing to 192.168.206.101: rc = -22 [ 1108.817091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1108.869396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1108.913872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1108.962880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1109.003148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1109.041403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1109.079832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1109.113114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1109.129927] LNet: 66320:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1109.146463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1109.184745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1109.239313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1109.290846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1109.330880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1109.365282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1109.714636] LNet: 66359:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1109.734463] LNetError: 66359:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1109.755413] LNet: Removed LNI 192.168.206.1@tcp [ 1110.168530] Key type .llcrypt unregistered [ 1110.171557] Key type ._llcrypt unregistered [ 1128.851174] Key type ._llcrypt registered [ 1128.853900] Key type .llcrypt registered [ 1129.156989] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1129.172875] alg: No test for adler32 (adler32-zlib) [ 1130.147255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1130.197493] LNet: Added LNI 192.168.206.1@tcp [8/256/0/180] [ 1138.059713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.101@tcp [ 1147.262391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1147.324665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1147.362696] LNet: Added LNI 192.168.206.1@tcp1 [8/256/0/180] [ 1148.704583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.1@tcp -d 192.168.206.101@tcp -m GET -e network_timeout -r 1 [ 1148.745313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.1@tcp -d 192.168.206.101@tcp1 -m GET -e network_timeout -r 1 [ 1148.789993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.1@tcp -d 192.168.206.1@tcp -m GET -e network_timeout -r 1 [ 1148.859688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.1@tcp -d 192.168.206.1@tcp1 -m GET -e network_timeout -r 1 [ 1148.911538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.1@tcp1 -d 192.168.206.101@tcp -m GET -e network_timeout -r 1 [ 1148.972891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.1@tcp1 -d 192.168.206.101@tcp1 -m GET -e network_timeout -r 1 [ 1149.019857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.1@tcp1 -d 192.168.206.1@tcp -m GET -e network_timeout -r 1 [ 1149.056754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.1@tcp1 -d 192.168.206.1@tcp1 -m GET -e network_timeout -r 1 [ 1149.087747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.101@tcp [ 1149.107983] LNet: There was an unexpected network error while writing to 192.168.206.101: rc = -22 [ 1149.415521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1149.482880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1149.538796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1149.611378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1149.656248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1149.664323] LNet: 1 local NIs in recovery (showing 1): 192.168.206.1@tcp [ 1149.667470] LNet: 1 peer NIs in recovery (showing 1): 192.168.206.101@tcp [ 1149.708602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1149.763175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1149.799044] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1149.834504] LNet: 67234:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1149.868080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1149.913687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1149.947619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1149.993450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1150.035883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1150.068422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1155.023586] LNet: There was an unexpected network error while writing to 192.168.206.101: rc = -22 [ 1155.402047] LNet: 67308:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1155.411979] LNetError: 67308:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1155.419820] LNet: Removed LNI 192.168.206.1@tcp [ 1155.440921] LNet: Removed LNI 192.168.206.1@tcp1 [ 1156.077145] Key type .llcrypt unregistered [ 1156.079961] Key type ._llcrypt unregistered [ 1175.723667] Key type ._llcrypt registered [ 1175.729894] Key type .llcrypt registered [ 1177.323412] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 00:35:05 (1788582905) [ 1177.845550] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1177.875755] alg: No test for adler32 (adler32-zlib) [ 1178.823774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1178.895785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1178.920406] LNet: Added LNI 192.168.206.1@tcp [8/256/0/180] [ 1179.793624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1179.813874] LNet: Added LNI 192.168.206.1@tcp1 [8/256/0/180] [ 1181.054940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.1@tcp [ 1181.129675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1181.161420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1181.197376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1181.236687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1181.264670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1181.312279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.206.1@tcp [ 1181.342373] LNetError: 68405:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.206.1@tcp to recovery [ 1181.920323] LNet: 1 local NIs in recovery (showing 1): 192.168.206.1@tcp [ 1181.920725] LNet: There was an unexpected network error while writing to 192.168.206.1: rc = -22 [ 1183.968368] LNet: There was an unexpected network error while writing to 192.168.206.1: rc = -22 [ 1184.895114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1187.044072] LNet: There was an unexpected network error while writing to 192.168.206.1: rc = -22 [ 1187.385688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1187.417374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1187.499903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1187.548317] LNet: 68502:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1187.554848] LNetError: 68502:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1187.561098] LNet: Removed LNI 192.168.206.1@tcp [ 1187.567887] LNet: Removed LNI 192.168.206.1@tcp1 [ 1187.592712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1187.648647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1187.705688] LNet: Added LNI 192.168.206.1@tcp [8/256/0/180] [ 1188.432822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1188.450660] LNet: Added LNI 192.168.206.1@tcp1 [8/256/0/180] [ 1189.255311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.1@tcp [ 1189.290815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1189.360967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1189.422988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1189.488853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1189.540037] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1189.585487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1189.637489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.206.1@tcp [ 1189.665422] LNetError: 68775:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.206.1@tcp to recovery [ 1189.677971] LNet: There was an unexpected network error while writing to 192.168.206.1: rc = -22 [ 1192.169441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1195.808442] LNet: There was an unexpected network error while writing to 192.168.206.1: rc = -22 [ 1195.817136] LNet: Skipped 1 previous similar message [ 1200.234796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1200.269462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1200.305861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1206.639257] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 00:35:35 (1788582935) [ 1207.034539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1207.060625] LNet: 69483:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1207.069267] LNetError: 69483:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1207.077089] LNetError: 69483:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1207.093567] LNet: Removed LNI 192.168.206.1@tcp [ 1207.117034] LNet: Removed LNI 192.168.206.1@tcp1 [ 1207.149104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1207.189264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1207.206559] LNet: Added LNI 192.168.206.1@tcp [8/256/0/180] [ 1207.988939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1208.016461] LNet: Added LNI 192.168.206.1@tcp1 [8/256/0/180] [ 1208.921316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.1@tcp [ 1208.995423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1209.044256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1209.094670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1209.173065] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1209.224396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1209.276513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.206.1@tcp --health 0 [ 1209.389743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1210.273062] LNet: 1 peer NIs in recovery (showing 1): 192.168.206.1@tcp [ 1210.280465] LNet: There was an unexpected network error while writing to 192.168.206.1: rc = -22 [ 1210.280474] LNet: Skipped 1 previous similar message [ 1224.987615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1225.132953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1225.181877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1225.227267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.206.1@tcp --health 500 [ 1225.316653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1232.045337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1232.173947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1232.215132] LNet: 69976:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1232.222279] LNetError: 69976:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1232.233280] LNetError: 69976:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1232.244709] LNet: Removed LNI 192.168.206.1@tcp [ 1232.266669] LNet: Removed LNI 192.168.206.1@tcp1 [ 1232.324684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1232.389736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1232.436584] LNet: Added LNI 192.168.206.1@tcp [8/256/0/180] [ 1233.227074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1233.246554] LNet: Added LNI 192.168.206.1@tcp1 [8/256/0/180] [ 1234.376578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.1@tcp [ 1234.449764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1234.574329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1234.638611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1234.715034] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1234.789818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1234.846542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1234.901535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.206.1@tcp --health 0 [ 1235.425220] LNet: There was an unexpected network error while writing to 192.168.206.1: rc = -22 [ 1235.438011] LNet: Skipped 3 previous similar messages [ 1236.371304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1246.738872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1246.774845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1246.812397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1253.115738] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 00:36:22 (1788582982) [ 1253.568243] LNet: 71023:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1253.572985] LNetError: 71023:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1253.577413] LNetError: 71023:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1254.624147] LNet: Removed LNI 192.168.206.1@tcp [ 1254.636066] LNet: Removed LNI 192.168.206.1@tcp1 [ 1255.227897] Key type .llcrypt unregistered [ 1255.231807] Key type ._llcrypt unregistered [ 1261.793828] Key type ._llcrypt registered [ 1261.803164] Key type .llcrypt registered [ 1262.034582] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1262.050671] alg: No test for adler32 (adler32-zlib) [ 1263.008942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1263.028920] LNet: Added LNI 192.168.206.1@tcp [8/256/0/180] [ 1272.134107] Lustre: DEBUG MARKER: Initial discovery [ 1272.319411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.206.101@tcp [ 1274.776972] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1274.969522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1275.021257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.206.101@tcp [ 1275.054737] LNet: There was an unexpected network error while writing to 192.168.206.101: rc = -22 [ 1275.076390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1275.106873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.206.101@tcp -r 1 -m GET -l 3 [ 1275.133758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.206.101@tcp -r 1 -m GET -e local_error [ 1275.170466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.206.101@tcp -r 1 -m PUT -l 6 [ 1275.360520] LNet: 1 local NIs in recovery (showing 1): 192.168.206.1@tcp [ 1276.593271] Lustre: DEBUG MARKER: Force oleg601-client.virtnet to discover 192.168.206.101@tcp (in background) [ 1276.633164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.206.101@tcp [ 1278.061310] Lustre: DEBUG MARKER: Force oleg601-server to discover 192.168.206.1@tcp [ 1279.713095] LNet: There was an unexpected network error while writing to 192.168.206.101: rc = -22 [ 1280.417989] Lustre: DEBUG MARKER: Wait for 71775 [ 1282.002601] Lustre: DEBUG MARKER: Finished wait on 71775 [ 1282.062982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay del -a [ 1282.118080] LNet: There was an unexpected network error while writing to 192.168.206.101: rc = -22 [ 1282.126555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1282.351960] LNet: 72233:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1282.356394] LNetError: 72233:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1282.372808] LNet: Removed LNI 192.168.206.1@tcp [ 1282.865676] Key type .llcrypt unregistered [ 1282.871801] Key type ._llcrypt unregistered [ 1299.678158] Key type ._llcrypt registered [ 1299.680451] Key type .llcrypt registered [ 1301.243076] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 00:37:09 (1788583029) [ 1301.732715] Key type .llcrypt unregistered [ 1301.735913] Key type ._llcrypt unregistered [ 1308.335162] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1308.506628] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1308.511682] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1308.516123] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1311.108709] Key type ._llcrypt registered [ 1311.112451] Key type .llcrypt registered [ 1311.321199] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1311.336526] alg: No test for adler32 (adler32-zlib) [ 1312.327950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1312.460319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1312.517578] LNet: Added LNI 192.168.206.1@tcp [8/256/0/180] [ 1313.516659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1313.546744] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1321.402245] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 00:37:30 (1788583050) [ 1321.700928] LNet: 74301:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1321.708848] LNetError: 74301:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1321.719358] LNet: Removed LNI 10.1.2.3@tcp [ 1321.728115] LNet: Removed LNI 192.168.206.1@tcp [ 1322.290579] Key type .llcrypt unregistered [ 1322.294545] Key type ._llcrypt unregistered [ 1328.187913] Key type ._llcrypt registered [ 1328.191736] Key type .llcrypt registered [ 1328.344110] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1328.355887] alg: No test for adler32 (adler32-zlib) [ 1329.283132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1329.351866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1329.378950] LNet: Added LNI 192.168.206.1@tcp [8/256/0/180] [ 1329.966715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1329.992690] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1330.908989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.206.1@tcp [ 1332.001159] LNetError: Unexpected error -22 connecting to 192.168.206.1@tcp at host 192.168.206.1:988 [ 1337.616467] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 00:37:46 (1788583066) [ 1337.912923] LNet: 75431:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1337.935617] LNetError: 75431:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1337.943972] LNet: Removed LNI 10.1.2.3@tcp [ 1337.959451] LNet: Removed LNI 192.168.206.1@tcp [ 1338.731184] Key type .llcrypt unregistered [ 1338.740348] Key type ._llcrypt unregistered [ 1345.082599] Key type ._llcrypt registered [ 1345.086803] Key type .llcrypt registered [ 1345.346978] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1345.366644] alg: No test for adler32 (adler32-zlib) [ 1346.257734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1346.304549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1346.328459] LNet: Added LNI 192.168.206.1@tcp1 [8/256/0/180] [ 1347.177268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1347.221814] LNet: Added LNI 192.168.206.1@tcp2 [8/256/0/180] [ 1348.306780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.206.1@tcp1 --nid 192.168.206.1@tcp2 [ 1351.987371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.1@tcp1 192.168.206.1@tcp2 [ 1352.002978] LNetError: 76264:0:(lib-move.c:2101:lnet_handle_find_routed_path()) Unable to send message from 192.168.206.1@tcp1 to 192.168.206.1@tcp2 - Route table may be misconfigured [ 1352.022807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.1@tcp2 192.168.206.1@tcp1 [ 1357.428828] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 00:38:06 (1788583086) [ 1357.642283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1357.662896] LNet: 76837:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1357.668539] LNetError: 76837:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1357.681630] LNet: Removed LNI 192.168.206.1@tcp1 [ 1358.714880] LNet: Removed LNI 192.168.206.1@tcp2 [ 1358.751982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1358.824583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1358.853584] LNet: Added LNI 192.168.206.1@tcp [8/256/0/180] [ 1359.447915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1359.479124] LNet: Added LNI 192.168.206.1@tcp1 [8/256/0/180] [ 1360.245782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.1@tcp [ 1360.300564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.1@tcp 192.168.206.1@tcp [ 1360.328965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.1@tcp1 192.168.206.1@tcp1 [ 1360.375346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.206.1@tcp -d 192.168.206.1@tcp -e network_timeout [ 1360.411863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.206.1@tcp -d 192.168.206.1@tcp1 -e network_timeout [ 1360.448685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.206.1@tcp1 -d 192.168.206.1@tcp -e network_timeout [ 1360.491554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.206.1@tcp1 -d 192.168.206.1@tcp1 -e network_timeout [ 1360.563465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.1@tcp [ 1360.589960] LNet: There was an unexpected network error while writing to 192.168.206.1: rc = -22 [ 1360.611255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1360.709171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1360.824597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1360.864925] LNet: 1 local NIs in recovery (showing 1): 192.168.206.1@tcp [ 1366.163742] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 00:38:15 (1788583095) [ 1366.263635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1366.296186] LNet: 77694:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1366.306418] LNetError: 77694:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1366.312562] LNetError: 77694:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1366.318098] LNet: Removed LNI 192.168.206.1@tcp [ 1367.337995] LNet: Removed LNI 192.168.206.1@tcp1 [ 1367.359448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1367.447885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1367.765494] LNet: 77735:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1368.119511] Key type .llcrypt unregistered [ 1368.123250] Key type ._llcrypt unregistered [ 1377.832517] Key type ._llcrypt registered [ 1377.834636] Key type .llcrypt registered [ 1378.930459] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 00:38:27 (1788583107) [ 1379.250648] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1379.265684] alg: No test for adler32 (adler32-zlib) [ 1380.171338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1381.433087] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1382.774608] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 00:38:31 (1788583111) [ 1382.841659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1382.865554] LNet: 78824:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1382.905088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1382.998266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1383.038377] LNet: Added LNI 192.168.206.1@tcp [8/256/0/180] [ 1383.684769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1383.698664] LNet: Added LNI 192.168.206.1@tcp1 [8/256/0/180] [ 1384.318137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.1@tcp [ 1384.344516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.1@tcp1 [ 1384.370324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.1@tcp1 [ 1389.970264] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 00:38:38 (1788583118) [ 1391.571933] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg601-server [ 1393.013967] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 00:38:42 (1788583122) [ 1394.586945] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg601-server [ 1396.255406] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 00:38:45 (1788583125) [ 1398.131285] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg601-server [ 1400.301823] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 00:38:49 (1788583129) [ 1402.016328] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg601-server [ 1403.674676] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 00:38:52 (1788583132) [ 1405.279582] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg601-server [ 1406.762068] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 00:38:55 (1788583135) [ 1408.658603] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg601-server [ 1410.395692] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 00:38:59 (1788583139) [ 1412.442443] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg601-server [ 1414.415723] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 00:39:03 (1788583143) [ 1416.186909] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg601-server [ 1418.130276] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 00:39:06 (1788583146) [ 1419.759759] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg601-server [ 1421.467560] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 00:39:10 (1788583150) [ 1421.533022] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1421.550127] LNet: 82700:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1421.555638] LNetError: 82700:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1421.568693] LNet: Removed LNI 192.168.206.1@tcp [ 1422.631672] LNet: Removed LNI 192.168.206.1@tcp1 [ 1422.670718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1422.732463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1422.761213] LNet: Added LNI 192.168.206.1@tcp [8/256/0/180] [ 1423.299752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1423.424417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.1@tcp [ 1423.639877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1423.662649] LNet: 82853:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1423.667337] LNetError: 82853:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1423.672436] LNetError: 82853:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1423.688714] LNet: Removed LNI 192.168.206.1@tcp [ 1423.720399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1423.798676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1423.835549] LNet: Added LNI 192.168.206.1@tcp [8/256/0/180] [ 1424.552530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.1@tcp [ 1424.686200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1424.744340] LNetError: 82998:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1424.758653] LNet: Removed LNI 192.168.206.1@tcp [ 1424.775573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1424.836766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1424.857237] LNet: Added LNI 192.168.206.1@tcp [8/256/0/180] [ 1425.543532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1425.620743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.1@tcp [ 1425.852240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1425.876231] LNet: 83146:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1425.880407] LNet: 83146:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1426.928259] LNet: Removed LNI 192.168.206.1@tcp [ 1426.976528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1427.077974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1427.159250] LNet: Added LNI 192.168.206.1@tcp [8/256/0/180] [ 1428.230549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.1@tcp [ 1428.451162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1428.478245] LNetError: 83291:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1428.484040] LNetError: 83291:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1429.559372] LNet: Removed LNI 192.168.206.1@tcp [ 1429.602201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1429.701621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1429.734468] LNet: Added LNI 192.168.206.1@tcp [8/256/0/180] [ 1430.628833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1430.738684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.1@tcp [ 1431.048944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1431.080528] LNet: 83438:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1431.087226] LNet: 83438:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1432.103343] LNet: Removed LNI 192.168.206.1@tcp [ 1432.167512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1432.284281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1432.341791] LNet: Added LNI 192.168.206.1@tcp [8/256/0/180] [ 1433.154552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.1@tcp [ 1433.368478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1433.401880] LNetError: 83584:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1433.409197] LNetError: 83584:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1433.421479] LNet: Removed LNI 192.168.206.1@tcp [ 1433.447206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1433.509460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1433.526057] LNet: Added LNI 192.168.206.1@tcp [8/256/0/180] [ 1434.189933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1434.277934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.1@tcp [ 1435.751771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1436.841697] LNet: Removed LNI 192.168.206.1@tcp [ 1436.904371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1437.037288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1437.096366] LNet: Added LNI 192.168.206.1@tcp [8/256/0/180] [ 1437.933343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.1@tcp [ 1438.209342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1439.272264] LNet: Removed LNI 192.168.206.1@tcp [ 1439.302866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1439.353558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1439.378169] LNet: Added LNI 192.168.206.1@tcp [8/256/0/180] [ 1440.056602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1440.130661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.1@tcp [ 1440.430252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1440.463804] LNet: 84033:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1440.478210] LNet: 84033:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 1441.513163] LNet: Removed LNI 192.168.206.1@tcp [ 1441.564343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1441.630206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1441.680902] LNet: Added LNI 192.168.206.1@tcp [8/256/0/180] [ 1442.373917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.1@tcp [ 1442.693900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1442.737287] LNetError: 84178:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1442.742750] LNetError: 84178:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 1442.762084] LNet: Removed LNI 192.168.206.1@tcp [ 1442.798281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1442.888533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1442.921314] LNet: Added LNI 192.168.206.1@tcp [8/256/0/180] [ 1443.711356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1443.789097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.1@tcp [ 1444.102417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1445.166512] LNet: Removed LNI 192.168.206.1@tcp [ 1445.214136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1445.326259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1445.354883] LNet: Added LNI 192.168.206.1@tcp [8/256/0/180] [ 1446.252420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.1@tcp [ 1446.573366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1447.656405] LNet: Removed LNI 192.168.206.1@tcp [ 1447.685766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1447.758483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1447.798212] LNet: Added LNI 192.168.206.1@tcp [8/256/0/180] [ 1448.670134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1448.727327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1448.749357] LNet: Removed LNI 192.168.206.1@tcp [ 1448.770518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1448.824870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1448.851498] LNet: Added LNI 192.168.206.1@tcp [8/256/0/180] [ 1449.486463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1449.549755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1449.571761] LNet: Removed LNI 192.168.206.1@tcp [ 1449.591302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1449.629842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 128 [ 1449.719978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1449.764179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1449.822171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 1000000000000000000000000 [ 1449.905168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1449.951550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1450.045043] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer foo [ 1450.118566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1450.157506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1450.240598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 8 [ 1450.269314] LNet: Added LNI 192.168.206.1@tcp [8/256/0/180] [ 1450.964572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1450.982800] LNet: Removed LNI 192.168.206.1@tcp [ 1450.998126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1451.056968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1457.209879] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 00:39:46 (1788583186) [ 1457.494956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1457.513861] LNet: 85492:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1457.519713] LNet: 85492:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 9 previous similar messages [ 1457.547557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1457.634453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1457.692487] LNet: Added LNI 192.168.206.1@tcp231 [8/256/0/180] [ 1458.692143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1458.726991] LNet: Removed LNI 192.168.206.1@tcp231 [ 1458.749496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1458.816982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1458.839060] LNet: Added LNI 192.168.206.1@tcp231 [8/256/0/180] [ 1458.844119] LNet: 85638:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1459.128893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1459.158733] LNet: Removed LNI 192.168.206.1@tcp231 [ 1459.182733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1459.219113] LNet: Added LNI 192.168.206.1@tcp231 [8/256/0/180] [ 1460.482869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1460.500461] LNetError: 85783:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1460.511948] LNetError: 85783:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 6 previous similar messages [ 1460.526727] LNet: Removed LNI 192.168.206.1@tcp231 [ 1460.556120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1460.637680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1460.661396] LNet: Added LNI 192.168.206.1@tcp231 [8/256/0/0] [ 1460.668065] LNet: 85791:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1461.024496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1461.068831] LNet: Removed LNI 192.168.206.1@tcp231 [ 1461.082455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1461.109688] LNet: Added LNI 192.168.206.1@tcp231 [8/256/0/0] [ 1462.364217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1462.404660] LNet: Removed LNI 192.168.206.1@tcp231 [ 1462.463219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1462.532579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1462.557475] LNet: Added LNI 192.168.206.1@tcp231 [8/256/0/60] [ 1462.562614] LNet: 85942:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1463.023599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1463.049188] LNet: Removed LNI 192.168.206.1@tcp231 [ 1463.072412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1463.093840] LNet: Added LNI 192.168.206.1@tcp231 [8/256/0/60] [ 1464.065780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1464.088707] LNet: Removed LNI 192.168.206.1@tcp231 [ 1464.110775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1464.218220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1464.242649] LNet: Added LNI 192.168.206.1@tcp231 [8/256/0/180] [ 1464.731803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1464.761509] LNet: Removed LNI 192.168.206.1@tcp231 [ 1464.777581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1464.816660] LNet: Added LNI 192.168.206.1@tcp231 [8/256/0/180] [ 1471.902748] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 00:40:00 (1788583200) [ 1472.219536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1472.259450] LNet: Removed LNI 192.168.206.1@tcp231 [ 1472.283547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1472.360137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1472.402845] LNet: Added LNI 192.168.206.1@tcp232 [8/256/0/180] [ 1473.106656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1473.155567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1473.212065] LNet: Removed LNI 192.168.206.1@tcp232 [ 1473.236699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1473.286052] LNet: Added LNI 192.168.206.1@tcp232 [8/256/0/180] [ 1474.022486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1474.047660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1474.092133] LNet: Removed LNI 192.168.206.1@tcp232 [ 1474.115188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1474.154214] LNet: Added LNI 192.168.206.1@tcp232 [8/256/0/180] [ 1474.764623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1474.810930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1474.844748] LNet: Removed LNI 192.168.206.1@tcp232 [ 1474.878807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1474.910961] LNet: Added LNI 192.168.206.1@tcp232 [8/256/0/180] [ 1475.661926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1475.692471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1475.721512] LNet: Removed LNI 192.168.206.1@tcp232 [ 1475.738183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1481.575357] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 00:40:10 (1788583210) [ 1482.544429] Key type .llcrypt unregistered [ 1482.547805] Key type ._llcrypt unregistered [ 1488.899196] Key type ._llcrypt registered [ 1488.901081] Key type .llcrypt registered [ 1489.118418] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1489.146801] alg: No test for adler32 (adler32-zlib) [ 1490.058225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1490.080085] LNet: Added LNI 192.168.206.1@tcp [8/256/0/180] [ 1497.470114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.101@tcp [ 1506.372656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1506.433736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1506.464016] LNet: Added LNI 192.168.206.1@tcp1 [8/256/0/180] [ 1507.736574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.1@tcp -d 192.168.206.101@tcp -m GET -e local_interrupt -r 2 [ 1507.772285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.1@tcp -d 192.168.206.101@tcp1 -m GET -e local_interrupt -r 2 [ 1507.811626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.1@tcp -d 192.168.206.1@tcp -m GET -e local_interrupt -r 2 [ 1507.863811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.1@tcp -d 192.168.206.1@tcp1 -m GET -e local_interrupt -r 2 [ 1507.899661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.1@tcp1 -d 192.168.206.101@tcp -m GET -e local_interrupt -r 2 [ 1507.952925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.1@tcp1 -d 192.168.206.101@tcp1 -m GET -e local_interrupt -r 2 [ 1507.982580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.1@tcp1 -d 192.168.206.1@tcp -m GET -e local_interrupt -r 2 [ 1508.011868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.1@tcp1 -d 192.168.206.1@tcp1 -m GET -e local_interrupt -r 2 [ 1508.041216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.1@tcp 192.168.206.101@tcp [ 1508.060606] LNet: There was an unexpected network error while writing to 192.168.206.101: rc = -22 [ 1508.069703] LNet: 1 local NIs in recovery (showing 1): 192.168.206.1@tcp [ 1508.297606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1508.324518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1508.364712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1508.398619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1508.427117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1508.452886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1508.493606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1508.537310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1508.571573] LNet: 88592:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1508.597874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1508.645188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1508.684555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1508.715397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1508.764339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1508.827934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1509.372431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.1@tcp -d 192.168.206.101@tcp -m GET -e local_dropped -r 2 [ 1509.421704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.1@tcp -d 192.168.206.101@tcp1 -m GET -e local_dropped -r 2 [ 1509.468705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.1@tcp -d 192.168.206.1@tcp -m GET -e local_dropped -r 2 [ 1509.541872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.1@tcp -d 192.168.206.1@tcp1 -m GET -e local_dropped -r 2 [ 1509.584856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.1@tcp1 -d 192.168.206.101@tcp -m GET -e local_dropped -r 2 [ 1509.653689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.1@tcp1 -d 192.168.206.101@tcp1 -m GET -e local_dropped -r 2 [ 1509.720553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.1@tcp1 -d 192.168.206.1@tcp -m GET -e local_dropped -r 2 [ 1509.778582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.1@tcp1 -d 192.168.206.1@tcp1 -m GET -e local_dropped -r 2 [ 1509.818094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.1@tcp 192.168.206.101@tcp [ 1509.848338] LNet: There was an unexpected network error while writing to 192.168.206.101: rc = -22 [ 1510.107792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1510.140864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1510.189613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1510.231973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1510.273915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1510.320758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1510.372733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1510.409518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1510.437252] LNet: 88713:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1510.452628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1510.489940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1510.523726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1510.545525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1510.572970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1510.596621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1511.092513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.1@tcp -d 192.168.206.101@tcp -m GET -e local_aborted -r 2 [ 1511.149696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.1@tcp -d 192.168.206.101@tcp1 -m GET -e local_aborted -r 2 [ 1511.205951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.1@tcp -d 192.168.206.1@tcp -m GET -e local_aborted -r 2 [ 1511.263638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.1@tcp -d 192.168.206.1@tcp1 -m GET -e local_aborted -r 2 [ 1511.330546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.1@tcp1 -d 192.168.206.101@tcp -m GET -e local_aborted -r 2 [ 1511.359726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.1@tcp1 -d 192.168.206.101@tcp1 -m GET -e local_aborted -r 2 [ 1511.388959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.1@tcp1 -d 192.168.206.1@tcp -m GET -e local_aborted -r 2 [ 1511.416589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.1@tcp1 -d 192.168.206.1@tcp1 -m GET -e local_aborted -r 2 [ 1511.441421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.1@tcp 192.168.206.101@tcp [ 1511.462553] LNet: There was an unexpected network error while writing to 192.168.206.101: rc = -22 [ 1511.672438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1511.695734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1511.721663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1511.751403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1511.773376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1511.803997] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1511.850652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1511.906986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1511.934698] LNet: 88833:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1511.970295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1512.023298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1512.074466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1512.105534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1512.156360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1512.184712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1512.648347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.1@tcp -d 192.168.206.101@tcp -m GET -e local_no_route -r 2 [ 1512.675995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.1@tcp -d 192.168.206.101@tcp1 -m GET -e local_no_route -r 2 [ 1512.705277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.1@tcp -d 192.168.206.1@tcp -m GET -e local_no_route -r 2 [ 1512.730746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.1@tcp -d 192.168.206.1@tcp1 -m GET -e local_no_route -r 2 [ 1512.763284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.1@tcp1 -d 192.168.206.101@tcp -m GET -e local_no_route -r 2 [ 1512.789941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.1@tcp1 -d 192.168.206.101@tcp1 -m GET -e local_no_route -r 2 [ 1512.822749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.1@tcp1 -d 192.168.206.1@tcp -m GET -e local_no_route -r 2 [ 1512.864767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.1@tcp1 -d 192.168.206.1@tcp1 -m GET -e local_no_route -r 2 [ 1512.903216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.1@tcp 192.168.206.101@tcp [ 1513.167508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1513.210363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1513.251794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1513.299219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1513.330610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1513.353526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1513.384168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1513.408472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1513.433901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1513.469484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1513.497147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1513.539442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1513.596923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1513.649379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1514.142963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.1@tcp -d 192.168.206.101@tcp -m GET -e local_timeout -r 2 [ 1514.189806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.1@tcp -d 192.168.206.101@tcp1 -m GET -e local_timeout -r 2 [ 1514.255956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.1@tcp -d 192.168.206.1@tcp -m GET -e local_timeout -r 2 [ 1514.293616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.1@tcp -d 192.168.206.1@tcp1 -m GET -e local_timeout -r 2 [ 1514.329937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.1@tcp1 -d 192.168.206.101@tcp -m GET -e local_timeout -r 2 [ 1514.366994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.1@tcp1 -d 192.168.206.101@tcp1 -m GET -e local_timeout -r 2 [ 1514.398637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.1@tcp1 -d 192.168.206.1@tcp -m GET -e local_timeout -r 2 [ 1514.438516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.1@tcp1 -d 192.168.206.1@tcp1 -m GET -e local_timeout -r 2 [ 1514.481194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.1@tcp 192.168.206.101@tcp [ 1514.502414] LNet: There was an unexpected network error while writing to 192.168.206.101: rc = -22 [ 1514.507368] LNet: Skipped 1 previous similar message [ 1514.773605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1514.801875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1514.827539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1514.861311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1514.896404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1514.930375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1515.010500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1515.039108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1515.056342] LNet: 89073:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1515.060585] LNet: 89073:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1515.086272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1515.117725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1515.170251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1515.222854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1515.251619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1515.292746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1515.848894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.1@tcp -d 192.168.206.101@tcp -m GET -e remote_dropped -r 2 [ 1515.893944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.1@tcp -d 192.168.206.101@tcp1 -m GET -e remote_dropped -r 2 [ 1515.929389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.1@tcp -d 192.168.206.1@tcp -m GET -e remote_dropped -r 2 [ 1515.961869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.1@tcp -d 192.168.206.1@tcp1 -m GET -e remote_dropped -r 2 [ 1515.986411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.1@tcp1 -d 192.168.206.101@tcp -m GET -e remote_dropped -r 2 [ 1516.014284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.1@tcp1 -d 192.168.206.101@tcp1 -m GET -e remote_dropped -r 2 [ 1516.078264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.1@tcp1 -d 192.168.206.1@tcp -m GET -e remote_dropped -r 2 [ 1516.142681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.1@tcp1 -d 192.168.206.1@tcp1 -m GET -e remote_dropped -r 2 [ 1516.186490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.1@tcp 192.168.206.101@tcp [ 1516.211843] LNet: 1 peer NIs in recovery (showing 1): 192.168.206.101@tcp [ 1516.477324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1516.517261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1516.564329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1516.617291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1516.651239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1516.694945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1516.731983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1516.764759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1516.798715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1516.831502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1516.857362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1516.892531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1516.923741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1516.981672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1521.758565] LNet: 89268:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1521.764334] LNetError: 89268:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1521.771073] LNet: Removed LNI 192.168.206.1@tcp [ 1521.792449] LNet: Removed LNI 192.168.206.1@tcp1 [ 1522.122739] Key type .llcrypt unregistered [ 1522.127273] Key type ._llcrypt unregistered [ 1538.758710] Key type ._llcrypt registered [ 1538.761318] Key type .llcrypt registered [ 1540.090891] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 00:41:08 (1788583268) [ 1540.626923] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1540.642875] alg: No test for adler32 (adler32-zlib) [ 1541.547603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1541.608608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1541.634022] LNet: Added LNI 192.168.206.1@tcp [8/256/0/180] [ 1542.434915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1542.494457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.206.1@tcp [ 1542.521420] LNetError: 90245:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.206.1@tcp to recovery [ 1542.614093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.206.1@tcp [ 1542.628629] LNet: 1 local NIs in recovery (showing 1): 192.168.206.1@tcp [ 1542.707887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1548.803539] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 00:41:17 (1788583277) [ 1548.992646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1549.025222] LNet: 90833:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1549.047555] LNetError: 90833:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1549.077772] LNet: Removed LNI 192.168.206.1@tcp [ 1549.110537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1549.163607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1549.195027] LNet: Added LNI 192.168.206.1@tcp [8/256/0/180] [ 1549.913985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1549.944390] LNet: Added LNI 192.168.206.1@tcp1 [8/256/0/180] [ 1551.088761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.206.1@tcp [ 1551.127520] LNetError: 91101:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.206.1@tcp to recovery [ 1551.203774] LNet: There was an unexpected network error while writing to 192.168.206.1: rc = -22 [ 1552.375209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.206.1@tcp [ 1560.089647] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 00:41:29 (1788583289) [ 1562.047978] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg601-server [ 1563.830830] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 00:41:32 (1788583292) [ 1565.775300] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg601-server [ 1567.507949] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 00:41:36 (1788583296) [ 1569.253545] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg601-server [ 1571.050691] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 00:41:39 (1788583299) [ 1571.134408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1571.164948] LNet: 92718:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1571.172379] LNetError: 92718:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1571.178525] LNet: Removed LNI 192.168.206.1@tcp [ 1571.190237] LNet: Removed LNI 192.168.206.1@tcp1 [ 1571.232943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1571.273826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1571.287650] LNet: Added LNI 192.168.206.1@tcp [8/256/0/180] [ 1571.927284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1571.982562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1572.013348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 1572.126795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1572.156640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 1572.248396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1579.083889] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 00:41:47 (1788583307) [ 1579.189464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1579.231499] LNet: 93449:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1579.243806] LNetError: 93449:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1579.250190] LNetError: 93449:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1579.283032] LNet: Removed LNI 192.168.206.1@tcp [ 1579.335351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1579.404553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1579.432922] LNet: Added LNI 192.168.206.1@tcp [8/256/0/180] [ 1587.692631] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 00:41:56 (1788583316) [ 1587.954946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1587.990732] LNet: 94157:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1587.996601] LNetError: 94157:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1588.011804] LNet: Removed LNI 192.168.206.1@tcp [ 1588.031604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1588.076282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1588.138689] LNet: Added LNI 192.168.206.1@tcp [8/256/0/180] [ 1595.546379] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 00:42:04 (1788583324) [ 1597.481080] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 1598.977846] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 00:42:07 (1788583327) [ 1599.465283] LNet: 95193:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1599.470803] LNetError: 95193:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1599.490330] LNet: Removed LNI 192.168.206.1@tcp [ 1600.083452] Key type .llcrypt unregistered [ 1600.086959] Key type ._llcrypt unregistered [ 1606.211493] Key type ._llcrypt registered [ 1606.214439] Key type .llcrypt registered [ 1606.405473] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1606.425450] alg: No test for adler32 (adler32-zlib) [ 1607.380652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1607.432489] LNet: Added LNI 192.168.206.1@tcp [8/256/0/180] [ 1615.187641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.101@tcp [ 1619.825375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.206.101@tcp [ 1620.233864] LNet: 95607:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1620.242228] LNetError: 95607:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1620.256711] LNet: Removed LNI 192.168.206.1@tcp [ 1620.628236] Key type .llcrypt unregistered [ 1620.635774] Key type ._llcrypt unregistered [ 1631.637758] Key type ._llcrypt registered [ 1631.640487] Key type .llcrypt registered [ 1632.921504] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 1634.380697] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 1636.063455] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 00:42:44 (1788583364) [ 1638.000127] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 [ 1639.607374] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 00:42:48 (1788583368) [ 1641.663194] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg601-server [ 1643.363826] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 00:42:52 (1788583372) [ 1645.118861] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg601-server [ 1646.837982] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 00:42:55 (1788583375) [ 1648.279227] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1648.448530] alg: No test for adler32 (adler32-zlib) [ 1649.554900] Lustre: Lustre: Build Version: 2.17.57_104_g4a8f637 [ 1649.828880] LNet: Added LNI 192.168.206.1@tcp [8/256/0/180] [ 1651.497151] Key type lgssc registered [ 1653.073865] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1673.816369] Key type lgssc unregistered [ 1674.121587] LNet: 98777:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1674.131497] LNetError: 98777:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1674.149786] LNet: Removed LNI 192.168.206.1@tcp [ 1674.930288] Key type .llcrypt unregistered [ 1674.935127] Key type ._llcrypt unregistered [ 1689.380495] Key type ._llcrypt registered [ 1689.383471] Key type .llcrypt registered [ 1690.976479] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 00:43:39 (1788583419) [ 1691.367372] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1691.410231] alg: No test for adler32 (adler32-zlib) [ 1692.368761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1692.461864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1692.503723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1692.601461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1692.665717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1700.559660] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 00:43:49 (1788583429) [ 1700.764937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1700.785760] LNet: 100190:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1700.810178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1700.876128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1700.901910] LNet: Added LNI 192.168.206.1@tcp [8/256/0/180] [ 1701.702596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1701.724901] LNet: 100334:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1701.735598] LNetError: 100334:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1701.754535] LNet: Removed LNI 192.168.206.1@tcp [ 1701.773418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1701.841175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1701.882367] LNet: Added LNI 192.168.206.1@tcp [8/256/0/180] [ 1702.622432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1702.667525] LNetError: 100362:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1702.684949] LNet: Removed LNI 192.168.206.1@tcp [ 1702.701813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1702.796272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1702.820256] LNet: Added LNI 192.168.206.1@tcp [8/256/0/180] [ 1710.073138] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 00:43:58 (1788583438) [ 1710.310840] LNet: 100951:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1710.326543] LNet: 100951:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1710.334813] LNetError: 100951:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1710.345980] LNet: Removed LNI 192.168.206.1@tcp [ 1711.061996] Key type .llcrypt unregistered [ 1711.065252] Key type ._llcrypt unregistered [ 1718.454283] Key type ._llcrypt registered [ 1718.457573] Key type .llcrypt registered [ 1718.629845] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1718.648923] alg: No test for adler32 (adler32-zlib) [ 1719.593818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1719.702525] LNet: Added LNI 192.168.206.1@tcp [8/256/0/180] [ 1719.713373] LNet: *** cfs_fail_loc=e003, val=100*** [ 1720.606993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.1@tcp [ 1720.632660] LNet: *** cfs_fail_loc=e003, val=100*** [ 1720.638311] LNet: Skipped 2 previous similar messages [ 1720.649302] Ping reply received with mapping: [ 1720.655698] found metadata reply NI[192.168.206.1@tcp] [ 1720.659376] 01 ce a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 1720.663583] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 1720.671369] 5a 5a 5a 5a 5a 5a 5a 5a [ 1720.834885] LNet: 101348:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1720.845373] LNetError: 101348:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1720.862370] LNet: Removed LNI 192.168.206.1@tcp [ 1721.642946] Key type .llcrypt unregistered [ 1721.647162] Key type ._llcrypt unregistered [ 1735.241290] Key type ._llcrypt registered [ 1735.243507] Key type .llcrypt registered [ 1736.506236] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 00:44:25 (1788583465) [ 1737.089207] Key type .llcrypt unregistered [ 1737.091901] Key type ._llcrypt unregistered [ 1743.138634] Key type ._llcrypt registered [ 1743.141792] Key type .llcrypt registered [ 1743.429103] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1743.448994] alg: No test for adler32 (adler32-zlib) [ 1744.491404] LNetError: 102385:0:(api-ni.c:2850:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 1744.870378] Key type .llcrypt unregistered [ 1744.879424] Key type ._llcrypt unregistered [ 1749.973038] Key type ._llcrypt registered [ 1749.975617] Key type .llcrypt registered [ 1751.288633] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 00:44:40 (1788583480) [ 1751.869381] Key type .llcrypt unregistered [ 1751.871685] Key type ._llcrypt unregistered [ 1758.925295] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1758.930415] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1761.591195] Key type ._llcrypt registered [ 1761.602540] Key type .llcrypt registered [ 1761.832971] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1761.848100] alg: No test for adler32 (adler32-zlib) [ 1762.808525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1762.853933] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1772.776937] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 00:45:01 (1788583501) [ 1772.922678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1772.953580] LNet: 104156:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1772.963762] LNetError: 104156:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1772.993168] LNet: Removed LNI 10.1.2.3@tcp [ 1773.026080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1773.649417] Key type .llcrypt unregistered [ 1773.654828] Key type ._llcrypt unregistered [ 1779.353772] Key type ._llcrypt registered [ 1779.358136] Key type .llcrypt registered [ 1779.518789] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1779.530099] alg: No test for adler32 (adler32-zlib) [ 1780.552304] LNet: Added LNI 192.168.206.1@tcp [8/256/0/180] [ 1780.663727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.1@tcp [ 1780.905844] LNet: 104443:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1780.913729] LNetError: 104443:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1781.930170] LNet: Removed LNI 192.168.206.1@tcp [ 1782.510432] Key type .llcrypt unregistered [ 1782.512142] Key type ._llcrypt unregistered [ 1793.703977] Key type ._llcrypt registered [ 1793.706130] Key type .llcrypt registered [ 1795.374203] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 00:45:23 (1788583523) [ 1796.102391] Key type .llcrypt unregistered [ 1796.104193] Key type ._llcrypt unregistered [ 1801.849685] Key type ._llcrypt registered [ 1801.852235] Key type .llcrypt registered [ 1802.059703] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1802.090616] alg: No test for adler32 (adler32-zlib) [ 1809.395182] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 00:45:38 (1788583538) [ 1809.528483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1809.580308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1809.684245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1809.743988] LNet: Added LNI 192.168.206.1@tcp [8/256/0/180] [ 1810.624092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1810.673937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 1810.717300] LNet: Removed LNI 192.168.206.1@tcp [ 1810.985468] LNet: 106181:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1811.514637] Key type .llcrypt unregistered [ 1811.519446] Key type ._llcrypt unregistered [ 1821.915583] Key type ._llcrypt registered [ 1821.922490] Key type .llcrypt registered [ 1823.206359] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 00:45:51 (1788583551) [ 1823.570283] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1823.592839] alg: No test for adler32 (adler32-zlib) [ 1824.497850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1824.544959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1824.561477] LNet: Added LNI 192.168.206.1@tcp [8/256/0/180] [ 1825.292899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.1@tcp [ 1825.321940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.206.1@tcp [ 1825.337543] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 1825.340301] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 1825.343295] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1825, 0, 0 [ 1825.350977] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 1825.353164] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 1825.356601] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 1825.360762] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1825, 0, 0 [ 1825.365523] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 1825.368237] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1825.371551] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 1825.374893] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1825, 0, 1875 [ 1825.382761] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 1825.386269] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1825.390235] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 1825.393505] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1825, 0, 1875 [ 1825.397044] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 1831.334900] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 00:46:00 (1788583560) [ 1831.810210] LNet: 107717:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1831.824724] LNetError: 107717:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1831.849220] LNet: Removed LNI 192.168.206.1@tcp [ 1832.460445] Key type .llcrypt unregistered [ 1832.466169] Key type ._llcrypt unregistered [ 1838.549221] Key type ._llcrypt registered [ 1838.552909] Key type .llcrypt registered [ 1838.852977] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1838.878099] alg: No test for adler32 (adler32-zlib) [ 1839.805281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1839.851976] LNet: Added LNI 192.168.206.1@tcp [8/256/0/180] [ 1846.220501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.101@tcp [ 1855.381603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1855.434243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1855.483867] LNet: Added LNI 192.168.206.1@tcp1 [8/256/0/180] [ 1856.361376] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1856.429938] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1856.443994] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1856.446518] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1858.868508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 1858.895342] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 1869.072843] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 00:46:37 (1788583597) [ 1869.495693] LNet: 109221:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1869.526327] LNetError: 109221:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1869.542108] LNet: Removed LNI 192.168.206.1@tcp [ 1870.631300] LNet: Removed LNI 192.168.206.1@tcp1 [ 1870.637257] LNetError: 109221:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1870.648611] LNetError: 109221:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1870.668917] LNet: Removed LNI 10.1.2.3@tcp99 [ 1871.493627] Key type .llcrypt unregistered [ 1871.499211] Key type ._llcrypt unregistered [ 1879.385947] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1879.393272] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1882.029804] Key type ._llcrypt registered [ 1882.032968] Key type .llcrypt registered [ 1882.280218] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1882.301271] alg: No test for adler32 (adler32-zlib) [ 1883.310525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1883.402635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1883.434901] LNet: Added LNI 192.168.206.1@tcp [8/256/0/180] [ 1884.236487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1884.268691] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1887.216983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 1887.284186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1887.359740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 1887.391835] LNetError: 109964:0:(peer.c:2362:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 1887.444860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1887.524873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.1@tcp [ 1888.114222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1888.175992] LNet: 109984:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1888.186773] LNetError: 109984:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1888.193680] LNet: Removed LNI 10.1.2.3@tcp [ 1889.256975] LNet: Removed LNI 192.168.206.1@tcp [ 1890.118488] Key type .llcrypt unregistered [ 1890.125336] Key type ._llcrypt unregistered [ 1906.601890] Key type ._llcrypt registered [ 1906.605192] Key type .llcrypt registered [ 1908.396882] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 00:47:16 (1788583636) [ 1909.079582] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1909.105805] alg: No test for adler32 (adler32-zlib) [ 1910.218855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1910.313860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1910.345590] LNet: Added LNI 192.168.206.1@tcp [8/256/0/180] [ 1911.511408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.1@tcp [ 1911.674783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg601-client@tcp [ 1920.185087] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 00:47:28 (1788583648) [ 1920.316642] LNet: 111507:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1920.328629] LNetError: 111507:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1920.362292] LNet: Removed LNI 192.168.206.1@tcp [ 1921.446474] Key type .llcrypt unregistered [ 1921.449491] Key type ._llcrypt unregistered [ 1929.145114] Key type ._llcrypt registered [ 1929.147759] Key type .llcrypt registered [ 1929.366670] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1929.383041] alg: No test for adler32 (adler32-zlib) [ 1930.375697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 1930.403572] LNet: Added LNI 192.168.206.1@tcp [8/256/0/180] [ 1930.426391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 1937.848972] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 00:47:46 (1788583666) [ 1939.654775] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 1942.020342] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 00:47:50 (1788583670) [ 1944.421803] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 1947.093888] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 00:47:55 (1788583675) [ 1947.244314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1947.281722] LNet: 112931:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1947.287951] LNetError: 112931:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1947.314969] LNet: Removed LNI 192.168.206.1@tcp [ 1947.354619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1947.532826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1947.577206] LNet: Added LNI 192.168.206.1@tcp [8/256/0/180] [ 1948.729703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 1948.820475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 1957.342363] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 00:48:06 (1788583686) [ 1957.770392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1957.820292] LNet: 113646:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1957.828125] LNetError: 113646:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1957.863222] LNet: Removed LNI 192.168.206.1@tcp [ 1957.897453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1957.992242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 1958.059320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1958.952512] Key type .llcrypt unregistered [ 1958.958840] Key type ._llcrypt unregistered [ 1972.507582] Key type ._llcrypt registered [ 1972.510416] Key type .llcrypt registered [ 1974.051352] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 00:48:22 (1788583702) [ 1974.595647] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1974.623263] alg: No test for adler32 (adler32-zlib) [ 1975.665909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1975.797919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1975.835558] LNet: Added LNI 192.168.206.1@tcp [8/256/0/180] [ 1976.715332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1976.779865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.1@tcp [ 1983.609032] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 00:48:32 (1788583712) [ 1983.725947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1983.759983] LNet: 115193:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1983.769449] LNetError: 115193:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1983.784546] LNet: Removed LNI 192.168.206.1@tcp [ 1983.824590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1983.918066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 1983.991861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 1992.240957] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 00:48:40 (1788583720) [ 1992.344876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1992.392491] LNet: 115772:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1992.451880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1992.566760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 1993.008821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1993.055602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1993.134976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 1993.646630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1993.691624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1993.755570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1993.786320] LNet: 115813:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1993.833581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1993.892419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1994.257112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1994.286234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1994.355741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 1994.991408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1995.042841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2001.889622] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 00:48:50 (1788583730) [ 2002.046678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2002.082367] LNet: 116412:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2002.116539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2002.196319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2002.212609] LNet: Added LNI 192.168.206.1@tcp [8/256/0/180] [ 2002.882427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2002.904474] LNet: Added LNI 192.168.206.1@tcp2 [8/256/0/180] [ 2003.850600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2003.913899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.1@tcp [ 2004.153252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2004.186421] LNetError: 116693:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2005.216211] LNet: Removed LNI 192.168.206.1@tcp [ 2005.229829] LNetError: 116693:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2005.255938] LNet: Removed LNI 192.168.206.1@tcp2 [ 2005.328242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2005.458951] LNet: Added LNI 192.168.206.1@tcp [8/256/0/180] [ 2005.470509] LNet: Added LNI 192.168.206.1@tcp2 [8/256/0/180] [ 2005.477544] LNet: 116699:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2006.387314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.1@tcp [ 2006.810640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2006.853900] LNet: 116737:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2006.860577] LNet: 116737:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 2007.904341] LNet: Removed LNI 192.168.206.1@tcp [ 2007.909625] LNetError: 116737:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2007.920677] LNetError: 116737:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2007.938350] LNet: Removed LNI 192.168.206.1@tcp2 [ 2007.964861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2008.076369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2008.100473] LNet: Added LNI 192.168.206.1@tcp [8/256/0/180] [ 2009.260568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2009.279524] LNet: Added LNI 192.168.206.1@tcp2 [8/256/0/180] [ 2010.242839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.1@tcp [ 2010.273114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2010.328231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.1@tcp [ 2010.555664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2011.616743] LNet: Removed LNI 192.168.206.1@tcp [ 2012.647909] LNet: Removed LNI 192.168.206.1@tcp2 [ 2012.680703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2012.810397] LNet: Added LNI 192.168.206.1@tcp [8/256/0/180] [ 2012.821851] LNet: Added LNI 192.168.206.1@tcp2 [8/256/0/180] [ 2012.827989] LNet: 117028:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2013.768120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.1@tcp [ 2014.050651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2014.077963] LNetError: 117065:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2014.086970] LNetError: 117065:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 2014.098457] LNet: Removed LNI 192.168.206.1@tcp [ 2015.143713] LNet: Removed LNI 192.168.206.1@tcp2 [ 2015.182373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2015.263920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2015.283465] LNet: Added LNI 192.168.206.1@tcp [8/256/0/180] [ 2016.163941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2016.206416] LNet: Added LNI 192.168.206.1@tcp2 [8/256/0/180] [ 2017.297693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2017.371520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.1@tcp [ 2017.711815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2017.746423] LNet: 117345:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2017.754025] LNet: 117345:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 2018.784242] LNet: Removed LNI 192.168.206.1@tcp [ 2018.799543] LNet: Removed LNI 192.168.206.1@tcp2 [ 2018.841778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2018.958853] LNet: Added LNI 192.168.206.1@tcp [8/256/0/180] [ 2018.970553] LNet: Added LNI 192.168.206.1@tcp2 [8/256/0/180] [ 2018.977317] LNet: 117351:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2019.820101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.1@tcp [ 2020.070360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2021.152222] LNet: Removed LNI 192.168.206.1@tcp [ 2021.187405] LNet: Removed LNI 192.168.206.1@tcp2 [ 2021.212286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2021.266606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2021.306720] LNet: Added LNI 192.168.206.1@tcp [8/256/0/180] [ 2022.132737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2022.160660] LNet: Added LNI 192.168.206.1@tcp2 [8/256/0/180] [ 2023.237583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.1@tcp [ 2023.319335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2023.399764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.1@tcp [ 2023.643881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2023.670922] LNetError: 117673:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2023.683585] LNetError: 117673:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 2023.695539] LNet: Removed LNI 192.168.206.1@tcp [ 2024.749862] LNet: Removed LNI 192.168.206.1@tcp2 [ 2024.805435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2024.891237] LNet: Added LNI 192.168.206.1@tcp [8/256/0/180] [ 2024.907357] LNet: Added LNI 192.168.206.1@tcp2 [8/256/0/180] [ 2024.912540] LNet: 117679:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2025.339538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.1@tcp [ 2025.522822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2025.541729] LNet: Removed LNI 192.168.206.1@tcp [ 2026.602316] LNet: Removed LNI 192.168.206.1@tcp2 [ 2026.672388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2026.757714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2026.807087] LNet: Added LNI 192.168.206.1@tcp [8/256/0/180] [ 2027.727874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2027.764939] LNet: Added LNI 192.168.206.1@tcp2 [8/256/0/180] [ 2028.822559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2028.871967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2028.981903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.1@tcp [ 2029.615707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2030.688515] LNet: Removed LNI 192.168.206.1@tcp [ 2030.710119] LNet: Removed LNI 192.168.206.1@tcp2 [ 2030.806382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2030.907212] LNet: Added LNI 192.168.206.1@tcp [8/256/0/180] [ 2030.913671] LNet: Added LNI 192.168.206.1@tcp2 [8/256/0/180] [ 2030.921369] LNet: 118019:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2031.448944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.1@tcp [ 2031.795530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2032.865076] LNet: Removed LNI 192.168.206.1@tcp [ 2032.877410] LNet: Removed LNI 192.168.206.1@tcp2 [ 2032.924894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2033.001591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2033.022809] LNet: Added LNI 192.168.206.1@tcp [8/256/0/180] [ 2033.783190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2033.808638] LNet: Added LNI 192.168.206.1@tcp2 [8/256/0/180] [ 2034.586288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.1@tcp [ 2034.617259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2034.646685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2034.697414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.1@tcp [ 2034.982795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2034.999428] LNet: 118373:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2035.003363] LNet: 118373:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 5 previous similar messages [ 2036.065631] LNet: Removed LNI 192.168.206.1@tcp [ 2037.117360] LNet: Removed LNI 192.168.206.1@tcp2 [ 2037.182967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2037.267617] LNet: Added LNI 192.168.206.1@tcp [8/256/0/180] [ 2037.273981] LNet: Added LNI 192.168.206.1@tcp2 [8/256/0/180] [ 2038.034868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.1@tcp [ 2046.732667] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 2049.338592] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 00:49:37 (1788583777) [ 2049.466315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2049.523685] LNetError: 119138:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2049.535859] LNetError: 119138:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 9 previous similar messages [ 2049.547860] LNet: Removed LNI 192.168.206.1@tcp [ 2050.600540] LNet: Removed LNI 192.168.206.1@tcp2 [ 2050.706926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2051.108886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2051.145223] LNet: Added LNI 192.168.206.1@tcp [8/256/0/180] [ 2052.008445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2052.056088] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2053.259413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2053.283208] LNet: Removed LNI 10.1.2.3@tcp [ 2053.293129] LNet: Removed LNI 192.168.206.1@tcp [ 2053.311703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2053.417024] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 2053.461325] LNet: Added LNI 192.168.206.1@tcp [8/256/0/180] [ 2053.479863] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2054.006904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 2054.052238] LNet: Removed LNI 192.168.206.1@tcp [ 2054.063653] LNet: Removed LNI 10.1.2.3@tcp [ 2061.101217] Lustre: DEBUG MARKER: == sanity-lnet test 475: per-operation latency stats populate and reset ========================================================== 00:49:50 (1788583790) [ 2061.445468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2061.489849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2061.565316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2061.590307] LNet: Added LNI 192.168.206.1@tcp [8/256/0/180] [ 2062.309481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2062.362370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.1@tcp [ 2062.429624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.1@tcp [ 2062.487154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.1@tcp [ 2062.527980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.1@tcp [ 2062.558738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.1@tcp [ 2062.590850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.1@tcp [ 2062.630549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.1@tcp [ 2062.668680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.1@tcp [ 2062.725960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.1@tcp [ 2062.819544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.1@tcp [ 2063.011706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2069.275373] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 00:49:58 (1788583798) [ 2069.511191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2069.547946] LNet: 120756:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2069.552603] LNet: 120756:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 2070.634160] LNet: Removed LNI 192.168.206.1@tcp [ 2070.686965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2070.826033] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2070.908027] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2070.914809] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2070.922758] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2073.381833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2073.438773] LNet: Added LNI 192.168.206.1@tcp [8/256/0/180] [ 2074.152916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2074.195562] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2076.707869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.1@tcp [ 2076.744045] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 2076.809546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.206.1@tcp -m PUT -e local_timeout -r 1 [ 2076.877696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 2076.973067] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2076.975696] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2077.212407] LNetError: 120821:0:(lib-move.c:4801:lnet_parse()) 192.168.206.1@tcp, src 192.168.206.1@tcp: Dropping PUT (error -108 looking up sender) [ 2078.243500] LNet: Removed LNI 10.1.2.3@tcp [ 2079.272589] LNet: Removed LNI 192.168.206.1@tcp [ 2079.808740] Key type .llcrypt unregistered [ 2079.812961] Key type ._llcrypt unregistered [ 2093.599946] Key type ._llcrypt registered [ 2093.601580] Key type .llcrypt registered [ 2095.347423] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 00:50:23 (1788583823) [ 2096.075116] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2096.099592] alg: No test for adler32 (adler32-zlib) [ 2097.069438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2097.624867] LNet: 122130:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2098.560378] Key type .llcrypt unregistered [ 2098.564637] Key type ._llcrypt unregistered [ 2107.134296] Key type ._llcrypt registered [ 2107.136357] Key type .llcrypt registered [ 2107.398450] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2107.422596] alg: No test for adler32 (adler32-zlib) [ 2108.558441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2108.600551] LNet: Added LNI 192.168.206.1@tcp [8/256/0/180] [ 2116.835789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.101@tcp [ 2117.100431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 2117.601277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.206.101@tcp [ 2117.834348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2117.859211] LNet: 1 peer NIs in recovery (showing 1): 192.168.206.101@tcp [ 2117.891646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2117.922915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2117.945963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2117.970735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2117.996722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2118.026509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2118.067198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2118.087090] LNet: 122577:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2118.130795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2118.170205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2118.226976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2118.276424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2118.323676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2118.379801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2118.901744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2119.134754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2119.162832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2119.193033] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2119.226480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2119.257852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2119.284663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2119.313267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2119.340879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2119.357206] LNet: 122694:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2119.376089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2119.405910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2119.436916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2119.478588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2119.520245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2119.556407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2120.318193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.206.101@tcp [ 2120.627203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2120.675678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2120.725978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2120.774769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2120.820570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2120.847041] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2120.891380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2120.920809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2120.939661] LNet: 122812:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2120.954973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2120.990801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2121.021181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2121.061279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2121.106500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2121.139949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2121.942340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2122.251602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2122.287439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2122.318104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2122.359365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2122.403854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2122.436365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2122.471608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2122.515783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2122.547401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2122.593089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2122.638752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2122.680994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2122.718570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2122.757764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2123.177266] LNet: 122975:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2123.186449] LNetError: 122975:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2123.228560] LNet: Removed LNI 192.168.206.1@tcp [ 2123.656649] Key type .llcrypt unregistered [ 2123.658318] Key type ._llcrypt unregistered [ 2140.934504] Key type ._llcrypt registered [ 2140.937717] Key type .llcrypt registered [ 2142.251837] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 00:51:11 (1788583871) [ 2142.600319] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2142.614406] alg: No test for adler32 (adler32-zlib) [ 2143.561524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2144.064230] LNet: 123860:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2144.924696] Key type .llcrypt unregistered [ 2144.928109] Key type ._llcrypt unregistered [ 2151.204310] Key type ._llcrypt registered [ 2151.210487] Key type .llcrypt registered [ 2151.463604] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2151.486129] alg: No test for adler32 (adler32-zlib) [ 2152.373470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2152.411131] LNet: Added LNI 192.168.206.1@tcp [8/256/0/180] [ 2159.856836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.101@tcp [ 2168.939500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2169.008822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2169.029797] LNet: Added LNI 192.168.206.1@tcp1 [8/256/0/180] [ 2170.312300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 2171.113511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.206.101@tcp [ 2171.430462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2171.504271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2171.582688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2171.636572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2171.720924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2171.787433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2171.845713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2171.873083] LNet: 1 peer NIs in recovery (showing 1): 192.168.206.101@tcp [ 2171.959882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2172.004740] LNet: 124510:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2172.033931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2172.065415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2172.103744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2172.150321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2172.213569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2172.271312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2173.152885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2173.521462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2173.566304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2173.610568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2173.652278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2173.706421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2173.752230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2173.807383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2173.882516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2173.924446] LNet: 124627:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2173.957729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2174.011663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2174.065250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2174.138673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2174.178699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2174.218740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2175.119567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.206.101@tcp [ 2175.466234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2175.511173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2175.555700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2175.597160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2175.656389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2175.699658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2175.773687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2175.863801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2175.914292] LNet: 124744:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2175.949419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2176.012717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2176.065477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2176.105965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2176.191345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2176.282978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2177.177748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2177.458853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2177.508972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2177.559358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2177.601606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2177.649375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2177.685594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2177.751955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2177.800593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2177.838633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2177.876266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2177.941581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2177.992123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2178.041756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2178.084743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2183.780428] LNet: 124943:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2183.787792] LNetError: 124943:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2183.794471] LNet: Removed LNI 192.168.206.1@tcp [ 2183.805879] LNet: Removed LNI 192.168.206.1@tcp1 [ 2184.337514] Key type .llcrypt unregistered [ 2184.347679] Key type ._llcrypt unregistered [ 2201.984778] Key type ._llcrypt registered [ 2201.986223] Key type .llcrypt registered [ 2203.368818] Lustre: DEBUG MARKER: == sanity-lnet test 525: keep reachable peer on local net delete ========================================================== 00:52:12 (1788583932) [ 2203.965474] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2203.975836] alg: No test for adler32 (adler32-zlib) [ 2204.959799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2205.028083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2205.056984] LNet: Added LNI 192.168.206.1@tcp [8/256/0/180] [ 2205.846712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2205.878756] LNet: Added LNI 192.168.206.1@tcp2 [8/256/0/180] [ 2206.704898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.1@tcp [ 2206.775725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp [ 2207.840647] LNet: Removed LNI 192.168.206.1@tcp [ 2214.787359] Lustre: DEBUG MARKER: == sanity-lnet test 550: DD-off: keep route up when gw reachable via another local NI ========================================================== 00:52:23 (1788583943) [ 2216.580239] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_550 Need at least 2 local interfaces for multi-rail [ 2218.320434] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 00:52:27 (1788583947) [ 2226.154326] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 00:52:34 (1788583954) [ 2232.761317] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 00:52:41 (1788583961) [ 2239.114109] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 00:52:48 (1788583968) [ 2246.047328] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 00:52:54 (1788583974) [ 2253.476025] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 00:53:02 (1788583982) [ 2260.591494] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 00:53:09 (1788583989) [ 2267.927833] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 00:53:16 (1788583996) [ 2274.802032] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 00:53:23 (1788584003) [ 2281.258673] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 00:53:30 (1788584010) [ 2287.307570] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 00:53:36 (1788584016) [ 2294.782669] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 00:53:43 (1788584023) [ 2301.658944] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 00:53:50 (1788584030) [ 2307.615887] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 00:53:56 (1788584036) [ 2314.637457] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 00:54:03 (1788584043) [ 2322.326790] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 00:54:11 (1788584051) [ 2330.405924] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 00:54:19 (1788584059) [ 2338.390605] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 00:54:27 (1788584067) [ 2338.464609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2338.490552] LNet: 136627:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2338.495881] LNetError: 136627:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2338.513845] LNet: Removed LNI 192.168.206.1@tcp2 [ 2338.542776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2338.770412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2338.810135] LNet: Added LNI 192.168.206.1@tcp [8/256/0/180] [ 2340.215389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2340.249206] LNet: 136776:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2340.257246] LNetError: 136776:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2340.273952] LNet: Removed LNI 192.168.206.1@tcp [ 2340.293680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2340.381343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2340.413245] LNet: Added LNI 192.168.206.1@tcp [8/256/0/180] [ 2347.808676] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 00:54:36 (1788584076) [ 2354.637842] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 00:54:43 (1788584083) [ 2361.488713] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 00:54:50 (1788584090) [ 2367.437361] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 00:54:56 (1788584096) [ 2372.880575] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 00:55:02 (1788584102) [ 2379.575950] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 00:55:08 (1788584108) [ 2385.263851] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 00:55:14 (1788584114) [ 2391.239452] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 00:55:20 (1788584120) [ 2397.334761] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 00:55:26 (1788584126) [ 2403.992031] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 00:55:32 (1788584132) [ 2409.819968] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 00:55:38 (1788584138) [ 2415.295796] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 00:55:44 (1788584144) [ 2421.656701] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 2195 sec ========== 00:55:50 (1788584150) [ 2421.873933] LNet: 144244:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2421.879996] LNetError: 144244:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2421.919140] LNet: Removed LNI 192.168.206.1@tcp [ 2422.462424] Key type .llcrypt unregistered [ 2422.466604] Key type ._llcrypt unregistered [ 2433.588028] Key type ._llcrypt registered [ 2433.628349] Key type .llcrypt registered [ 2434.008680] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2434.018235] alg: No test for adler32 (adler32-zlib) [ 2435.092627] Lustre: Lustre: Build Version: 2.17.57_104_g4a8f637 [ 2435.332112] LNet: Added LNI 192.168.206.1@tcp [8/256/0/180] [ 2437.048186] Key type lgssc registered [ 2438.176318] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2501.518608] Lustre: Mounted lustre-client - version 2.17.57_104_g4a8f637 [ 2506.999784] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2515.612941] Lustre: Unmounted lustre-client [ 2553.174969] Key type lgssc unregistered [ 2553.482991] LNet: 147192:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2553.499173] LNetError: 147192:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2553.520117] LNet: Removed LNI 192.168.206.1@tcp [ 2554.108181] Key type .llcrypt unregistered [ 2554.111803] Key type ._llcrypt unregistered