[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 442423474 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.002299] x2apic enabled [ 0.003009] Switched APIC routing to physical x2apic. [ 0.004014] kvm-guest: setup PV IPIs [ 0.007594] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008018] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009011] pid_max: default: 32768 minimum: 301 [ 0.010125] LSM: Security Framework initializing [ 0.011108] Yama: becoming mindful. [ 0.012046] SELinux: Initializing. [ 0.013085] *** VALIDATE selinux *** [ 0.021566] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026524] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027145] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028126] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029104] *** VALIDATE tmpfs *** [ 0.030458] *** VALIDATE proc *** [ 0.032250] *** VALIDATE cgroup *** [ 0.033010] *** VALIDATE cgroup2 *** [ 0.034269] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.035147] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.036009] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.037026] Spectre V2 : User space: Vulnerable [ 0.038010] Speculative Store Bypass: Vulnerable [ 0.041799] debug: unmapping init [mem 0xffffffffaa459000-0xffffffffaa460fff] [ 0.043663] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.044735] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.045022] ... version: 2 [ 0.046014] ... bit width: 48 [ 0.047010] ... generic registers: 4 [ 0.048013] ... value mask: 0000ffffffffffff [ 0.049014] ... max period: 00007fffffffffff [ 0.050012] ... fixed-purpose events: 3 [ 0.051012] ... event mask: 000000070000000f [ 0.052274] rcu: Hierarchical SRCU implementation. [ 0.054336] smp: Bringing up secondary CPUs ... [ 0.055556] x86: Booting SMP configuration: [ 0.056032] .... node #0, CPUs: #1 #2 #3 [ 0.059444] smp: Brought up 1 node, 4 CPUs [ 0.061012] smpboot: Max logical packages: 1 [ 0.062020] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.125549] node 0 deferred pages initialised in 60ms [ 0.128107] devtmpfs: initialized [ 0.129294] x86/mm: Memory block size: 128MB [ 0.133296] gcov: version magic: 0x41383552 [ 0.137308] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.138077] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.139242] pinctrl core: initialized pinctrl subsystem [ 0.140174] [ 0.140641] ************************************************************* [ 0.141012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.142012] ** ** [ 0.143010] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.144014] ** ** [ 0.145017] ** This means that this kernel is built to expose internal ** [ 0.146014] ** IOMMU data structures, which may compromise security on ** [ 0.147012] ** your system. ** [ 0.148013] ** ** [ 0.149016] ** If you see this message and you are not debugging the ** [ 0.150013] ** kernel, report this immediately to your vendor! ** [ 0.151011] ** ** [ 0.152013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.153013] ************************************************************* [ 0.154630] NET: Registered protocol family 16 [ 0.155427] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.156060] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.157057] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.158445] cpuidle: using governor menu [ 0.160613] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.163472] PCI: Using configuration type 1 for base access [ 0.166117] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.175064] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.176020] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.178070] cryptd: max_cpu_qlen set to 1000 [ 0.180133] ACPI: Added _OSI(Module Device) [ 0.181012] ACPI: Added _OSI(Processor Device) [ 0.182007] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.183011] ACPI: Added _OSI(Processor Aggregator Device) [ 0.186857] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.188687] ACPI: Interpreter enabled [ 0.189056] ACPI: PM: (supports S0 S3 S4 S5) [ 0.190011] ACPI: Using IOAPIC for interrupt routing [ 0.191090] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.192388] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.200993] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.202029] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.203018] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.204089] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.206305] acpiphp: Slot [2] registered [ 0.207113] acpiphp: Slot [5] registered [ 0.208133] acpiphp: Slot [6] registered [ 0.209114] acpiphp: Slot [3] registered [ 0.210084] acpiphp: Slot [4] registered [ 0.211090] acpiphp: Slot [7] registered [ 0.212097] acpiphp: Slot [8] registered [ 0.213083] acpiphp: Slot [9] registered [ 0.214099] acpiphp: Slot [10] registered [ 0.215095] acpiphp: Slot [11] registered [ 0.216101] acpiphp: Slot [12] registered [ 0.217120] acpiphp: Slot [13] registered [ 0.218137] acpiphp: Slot [14] registered [ 0.219114] acpiphp: Slot [15] registered [ 0.220095] acpiphp: Slot [16] registered [ 0.221092] acpiphp: Slot [17] registered [ 0.222104] acpiphp: Slot [18] registered [ 0.223073] acpiphp: Slot [19] registered [ 0.224092] acpiphp: Slot [20] registered [ 0.225113] acpiphp: Slot [21] registered [ 0.226110] acpiphp: Slot [22] registered [ 0.227118] acpiphp: Slot [23] registered [ 0.228340] acpiphp: Slot [24] registered [ 0.229073] acpiphp: Slot [25] registered [ 0.230045] acpiphp: Slot [26] registered [ 0.231104] acpiphp: Slot [27] registered [ 0.233171] acpiphp: Slot [28] registered [ 0.235130] acpiphp: Slot [29] registered [ 0.237131] acpiphp: Slot [30] registered [ 0.239111] acpiphp: Slot [31] registered [ 0.241084] PCI host bridge to bus 0000:00 [ 0.242028] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.245019] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.246018] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.248017] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.251024] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.254027] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.256181] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.260049] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.263456] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.272610] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.277059] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.280018] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.282018] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.285029] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.288132] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.290929] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.294045] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.296711] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.299932] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.309976] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.314021] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.320188] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.327013] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.330971] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.344015] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.357754] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.363000] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.368014] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.387016] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.397502] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.400364] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.403446] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.405297] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.407235] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.412032] iommu: Default domain type: Passthrough [ 0.413424] SCSI subsystem initialized [ 0.414225] ACPI: bus type USB registered [ 0.415118] usbcore: registered new interface driver usbfs [ 0.418107] usbcore: registered new interface driver hub [ 0.420090] usbcore: registered new device driver usb [ 0.421161] pps_core: LinuxPPS API ver. 1 registered [ 0.423011] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.427061] PTP clock support registered [ 0.429147] EDAC MC: Ver: 3.0.0 [ 0.432129] PCI: Using ACPI for IRQ routing [ 0.433717] NetLabel: Initializing [ 0.435011] NetLabel: domain hash size = 128 [ 0.437127] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.440081] NetLabel: unlabeled traffic allowed by default [ 0.442133] vgaarb: loaded [ 0.444186] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.446013] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.452303] clocksource: Switched to clocksource kvm-clock [ 0.563544] VFS: Disk quotas dquot_6.6.0 [ 0.565068] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.567465] *** VALIDATE ramfs *** [ 0.568716] *** VALIDATE hugetlbfs *** [ 0.570236] pnp: PnP ACPI init [ 0.572488] pnp: PnP ACPI: found 6 devices [ 0.595371] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.598501] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.600195] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.601779] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.603597] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.605448] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.607613] NET: Registered protocol family 2 [ 0.609758] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.616232] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.620177] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.626160] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.629909] TCP: Hash tables configured (established 65536 bind 65536) [ 0.632933] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.636433] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.639164] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.642352] NET: Registered protocol family 1 [ 0.644959] RPC: Registered named UNIX socket transport module. [ 0.646748] RPC: Registered udp transport module. [ 0.648267] RPC: Registered tcp transport module. [ 0.649543] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.651540] NET: Registered protocol family 44 [ 0.653978] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.656328] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.658910] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.661270] PCI: CLS 0 bytes, default 64 [ 0.662867] Unpacking initramfs... [ 2.055043] debug: unmapping init [mem 0xffff91d8bcc64000-0xffff91d8bffcffff] [ 2.059450] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.061934] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.065077] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.561782] Initialise system trusted keyrings [ 2.564362] Key type blacklist registered [ 2.566332] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.575564] zbud: loaded [ 2.578682] *** VALIDATE nfs *** [ 2.580016] *** VALIDATE nfs4 *** [ 2.581787] pstore: using deflate compression [ 2.585849] Platform Keyring initialized [ 2.688032] NET: Registered protocol family 38 [ 2.689651] Key type asymmetric registered [ 2.691224] Asymmetric key parser 'x509' registered [ 2.693278] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.696596] io scheduler mq-deadline registered [ 2.698475] io scheduler kyber registered [ 2.700147] io scheduler bfq registered [ 2.702409] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.706281] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.709706] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.713371] ACPI: Power Button [PWRF] [ 2.719480] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.725938] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.743347] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.771965] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.803475] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.809801] Non-volatile memory driver v1.3 [ 2.812012] Linux agpgart interface v0.103 [ 2.842158] virtio_blk virtio1: [vda] 149952 512-byte logical blocks (76.8 MB/73.2 MiB) [ 2.844975] vda: detected capacity change from 0 to 76775424 [ 2.865054] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 2.868214] vdb: detected capacity change from 0 to 1073741824 [ 2.875182] libphy: Fixed MDIO Bus: probed [ 2.879657] usbcore: registered new interface driver usbserial_generic [ 2.882478] usbserial: USB Serial support registered for generic [ 2.884318] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 2.889366] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 2.891468] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 2.894342] mousedev: PS/2 mouse device common for all mice [ 2.897293] rtc_cmos 00:05: RTC can wake from S4 [ 2.899472] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 2.903501] rtc_cmos 00:05: registered as rtc0 [ 2.905601] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 2.909150] intel_pstate: CPU model not supported [ 2.909716] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 2.914501] hid: raw HID events driver (C) Jiri Kosina [ 2.915024] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 2.915872] usbcore: registered new interface driver usbhid [ 2.920739] usbhid: USB HID core driver [ 2.922175] drop_monitor: Initializing network drop monitor service [ 2.923974] Initializing XFRM netlink socket [ 2.925916] NET: Registered protocol family 10 [ 2.928832] Segment Routing with IPv6 [ 2.930219] NET: Registered protocol family 17 [ 2.933042] mpls_gso: MPLS GSO support [ 2.939086] RAS: Correctable Errors collector initialized. [ 2.941362] AVX version of gcm_enc/dec engaged. [ 2.943139] AES CTR mode by8 optimization enabled [ 3.015059] sched_clock: Marking stable (3015024825, 0)->(3997593249, -982568424) [ 3.018477] registered taskstats version 1 [ 3.020892] Loading compiled-in X.509 certificates [ 3.022929] zswap: loaded using pool lzo/zbud [ 3.048275] Key type big_key registered [ 3.062484] Key type encrypted registered [ 3.064225] ima: No TPM chip found, activating TPM-bypass! [ 3.066376] ima: Allocated hash algorithm: sha1 [ 3.068208] ima: No architecture policies found [ 3.070076] evm: Initialising EVM extended attributes: [ 3.072172] evm: security.selinux [ 3.073426] evm: security.ima [ 3.074451] evm: security.capability [ 3.075842] evm: HMAC attrs: 0x1 [ 3.078660] rtc_cmos 00:05: setting system clock to 2026-09-17 19:13:42 UTC (1789672422) [ 3.084781] debug: unmapping init [mem 0xffffffffab403000-0xffffffffab5fffff] [ 3.088022] debug: unmapping init [mem 0xffffffffaa182000-0xffffffffaa458fff] [ 3.096079] Write protecting the kernel read-only data: 28672k [ 3.099843] debug: unmapping init [mem 0xffffffffa8803000-0xffffffffa89fffff] [ 3.102708] debug: unmapping init [mem 0xffffffffa9114000-0xffffffffa91fffff] [ 3.131834] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.140498] systemd[1]: Detected virtualization kvm. [ 3.142392] systemd[1]: Detected architecture x86-64. [ 3.144132] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.169676] systemd[1]: No hostname configured. [ 3.171413] systemd[1]: Set hostname to . [ 3.173537] random: systemd: uninitialized urandom read (16 bytes read) [ 3.176121] systemd[1]: Initializing machine ID from random generator. [ 3.219037] random: ln: uninitialized urandom read (6 bytes read) [ 3.301369] random: systemd: uninitialized urandom read (16 bytes read) [ 3.303644] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 3.309082] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 3.313895] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Memstrack Anylazing Service. Starting Setup Virtual Console... [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Slices. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Initrd Root Device. Starting Create Volatile Files and Directories... Starting Apply Kernel Variables... [ OK ] Reached target Swap. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 3.878318] device-mapper: uevent: version 1.0.3 [ 3.880791] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 4.621323] virtio_net virtio0 ens2: renamed from eth0 [ 4.732547] scsi host0: ata_piix [ 4.752653] scsi host1: ata_piix [ 4.798080] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 4.801133] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 8.554551] dracut-initqueue[ 8.557058] random: fast init done [580]: RTNETLINK answers: File exists [ 9.586272] random: crng init done [ 9.594184] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ 10.821103] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Reached target Remote File Systems. [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Timers. [ OK ] Stopped target Initrd Default Target. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 13.374429] printk: systemd: 25 output lines suppressed due to ratelimiting [ 14.209088] SELinux: Disabled at runtime. [ 14.318576] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 14.349895] systemd[1]: Detected virtualization kvm. [ 14.352526] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 15.664291] systemd[1]: initrd-switch-root.service: Succeeded. [ 15.672222] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 15.686241] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 15.693434] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 15.699380] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 15.715835] systemd[1]: Starting Journal Service... Starting Journal Service... [ 15.736644] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. Starting Apply Kernel Variables... [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... Mounting POSIX Message Queue File System... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Mounting Kernel Debug File System... [ OK ] Created slice system-getty.slice. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Started Dispatch Password Requests to Console Directory Watch. Mounting Huge Pages File System... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on Process Core Dump Socket. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on initctl Compatibility Named Pipe. [ 16.291701] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Created slice system-sshd\x2dkeygen.slice. Starting Remount Root and Kernel File Systems... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. [ OK ] Reached target rpc_pipefs.target. [ OK ] Started Journal Service. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Huge Pages File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started udev Coldplug all Devices. [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Mounted /mnt. [ 17.116734] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 17.950466] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 18.023152] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 18.160740] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 18.194036] EDAC sbridge: Ver: 1.1.2 [ 21.459942] Key type dns_resolver registered [ 21.909445] NFS: Registering the id_resolver key type [ 21.911515] Key type id_resolver registered [ 21.913941] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Login Service... Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Reached target sshd-keygen.target. [ OK ] Started irqbalance daemon. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... [ OK ] Started GSSAPI Proxy Daemon. Starting Hostname Service... [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg658-client login: [ 45.204031] hrtimer: interrupt took 5015524 ns [ 87.956701] libcfs: loading out-of-tree module taints kernel. [ 88.102575] Key type ._llcrypt registered [ 88.113448] Key type .llcrypt registered [ 88.524026] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 88.537485] alg: No test for adler32 (adler32-zlib) [ 90.315624] Lustre: Lustre: Build Version: 2.17.58_39_g9373f0b [ 91.390168] LNet: Added LNI 192.168.206.58@tcp [8/256/0/180] [ 93.120529] Key type lgssc registered [ 94.677745] Lustre: Echo OBD driver; http://www.lustre.org/ [ 222.450897] Lustre: Mounted lustre-client - version 2.17.58_39_g9373f0b [ 227.278269] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 241.182416] Lustre: DEBUG MARKER: oleg658-client.virtnet: executing check_logdir /tmp/testlogs/ [ 246.582913] Lustre: DEBUG MARKER: oleg658-client.virtnet: executing yml_node [ 248.290166] Lustre: lustre-OST0000-osc-ffff91d90548e000: disconnect after 23s idle [ 251.696553] Lustre: DEBUG MARKER: Client: 2.17.58.39 [ 254.175782] Lustre: DEBUG MARKER: MDS: 2.17.58.39 [ 256.493692] Lustre: DEBUG MARKER: OSS: 2.17.58.39 [ 258.211837] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Thu Sep 17 15:17:56 EDT 2026 [ 274.648090] Lustre: Unmounted lustre-client [ 291.980415] Key type lgssc unregistered [ 292.245625] LNet: 8408:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 292.250848] LNetError: 8408:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 292.272954] LNet: Removed LNI 192.168.206.58@tcp [ 292.867756] Key type .llcrypt unregistered [ 292.869881] Key type ._llcrypt unregistered [ 319.742694] Key type ._llcrypt registered [ 319.745432] Key type .llcrypt registered [ 319.931945] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 319.952873] alg: No test for adler32 (adler32-zlib) [ 320.930532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 321.011763] LNet: Added LNI 192.168.206.58@tcp [8/256/0/180] [ 321.052883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 325.472296] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 325.702710] LNet: 9288:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 325.709460] LNetError: 9288:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 325.722221] LNet: Removed LNI 192.168.206.58@tcp [ 326.450562] Key type .llcrypt unregistered [ 326.452291] Key type ._llcrypt unregistered [ 332.637993] Key type ._llcrypt registered [ 332.639886] Key type .llcrypt registered [ 333.888921] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 15:19:11 (1789672751) [ 334.401969] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 334.419911] alg: No test for adler32 (adler32-zlib) [ 335.334880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 335.433845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 335.471735] LNet: 9703:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 335.499958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 335.572439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 335.591906] LNet: 9710:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 342.957075] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 15:19:20 (1789672760) [ 343.180589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 343.211748] LNet: 10289:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 343.289575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 343.414416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 350.186981] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 15:19:28 (1789672768) [ 350.274638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 350.309263] LNet: 10869:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 350.342212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 350.454299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 356.856818] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 15:19:35 (1789672775) [ 356.971400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 357.007775] LNet: 11452:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 357.040957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 357.174699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 363.733646] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 15:19:41 (1789672781) [ 363.870250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 363.903557] LNet: 12040:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 363.945508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 364.023275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 364.424848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 371.007497] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 15:19:48 (1789672788) [ 371.111595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 371.164687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 371.238786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 371.650228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 372.148984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 378.585593] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 15:19:56 (1789672796) [ 378.762583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 378.810492] LNet: 13231:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 378.824647] LNet: 13231:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 378.863348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 379.014191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 385.606536] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 15:20:03 (1789672803) [ 385.708663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 385.750809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 385.853551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 385.913664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 386.519817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 386.551376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 387.096644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 387.143896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 387.609089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 387.674152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 388.231208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 388.266831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 388.790941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 388.828399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 389.379906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 389.420744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 389.922637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 389.971614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 390.405645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 390.433345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 390.824835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 390.867790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 391.333184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 391.363901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 391.731026] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 391.782118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 392.137188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 392.183164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 397.998153] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 15:20:16 (1789672816) [ 398.189829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 398.209834] LNet: 14498:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 398.215153] LNet: 14498:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 398.232508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 398.307494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 398.337941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 404.317618] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 15:20:22 (1789672822) [ 404.589467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 404.629417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 404.679970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 404.718133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 411.149050] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 15:20:29 (1789672829) [ 411.205753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 411.262177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 411.337492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 411.376300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 418.078444] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 15:20:36 (1789672836) [ 418.481149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 418.537084] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 418.611331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 418.636390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 425.889570] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 15:20:43 (1789672843) [ 426.117973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 426.187599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 426.288938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 426.355040] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 433.303906] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 15:20:51 (1789672851) [ 433.407582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 433.425142] LNet: 17423:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 433.432993] LNet: 17423:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 433.463320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 433.566944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 433.625829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 440.044474] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 15:20:57 (1789672857) [ 440.149684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 440.195507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 440.278712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 440.835878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 447.708746] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 15:21:05 (1789672865) [ 447.812544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 447.851704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 448.039865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 448.242506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 454.293308] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 15:21:12 (1789672872) [ 454.446351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 454.508960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 454.616311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 454.798974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 460.765782] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 15:21:18 (1789672878) [ 460.840357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 460.870330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 461.356414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 461.604510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 467.235287] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 15:21:25 (1789672885) [ 467.404594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 467.485717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 467.548508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 467.612424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 473.111600] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 15:21:31 (1789672891) [ 473.315591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 473.352606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 473.471145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 473.516411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 479.756419] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 15:21:37 (1789672897) [ 479.864914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 479.909388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 480.000349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 486.433911] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 15:21:44 (1789672904) [ 486.557360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 486.611192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 486.762259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 492.741174] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 15:21:51 (1789672911) [ 492.831497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 492.881755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 492.977564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 493.366517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 499.280840] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 15:21:57 (1789672917) [ 499.450255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 499.462889] LNet: 23456:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 499.467339] LNet: 23456:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 9 previous similar messages [ 499.479601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 499.534349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 499.565241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 504.966174] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 15:22:03 (1789672923) [ 505.105794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 505.155680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 505.256964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 505.292944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 511.285556] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 15:22:09 (1789672929) [ 511.391797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 511.461145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 511.514558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 511.540790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 518.751258] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 15:22:16 (1789672936) [ 518.915745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 518.954047] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 519.044935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 519.077330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 525.722956] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 15:22:23 (1789672943) [ 526.190205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 526.246913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 526.334053] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 526.447585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 526.502442] LNetError: 25813:0:(peer.c:2366:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 526.546235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 526.597722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 532.715183] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 15:22:31 (1789672951) [ 532.849578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 532.938394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 533.060548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.zgR86vxJD5 [ 533.142168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.zgR86vxJD5 [ 533.210978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.zgR86vxJD5 [ 539.295517] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 15:22:37 (1789672957) [ 539.401293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 539.485481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 539.541621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 539.592688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 539.622313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 545.716367] Lustre: DEBUG MARKER: == sanity-lnet test 40: LNetAddPeer merges a pre-existing non-primary peer NID ========================================================== 15:22:43 (1789672963) [ 545.958702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 546.048338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 546.122859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 546.154831] LNet: Added LNI 192.168.206.58@tcp [8/256/0/180] [ 547.389983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.58@tcp [ 548.733479] Lustre: Lustre: Build Version: 2.17.58_39_g9373f0b [ 550.776743] Key type lgssc registered [ 551.807493] Lustre: Echo OBD driver; http://www.lustre.org/ [ 555.095167] LNetError: 27574:0:(lib-move.c:2151:lnet_handle_find_routed_path()) peer 192.168.206.58@tcp99 has no available nets [ 557.630537] Key type lgssc unregistered [ 557.974844] LNetError: 28677:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 557.994158] LNet: Removed LNI 192.168.206.58@tcp [ 558.668257] Key type .llcrypt unregistered [ 558.671359] Key type ._llcrypt unregistered [ 564.942752] Key type ._llcrypt registered [ 564.944779] Key type .llcrypt registered [ 566.104297] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 15:23:04 (1789672984) [ 566.571402] Key type .llcrypt unregistered [ 566.575243] Key type ._llcrypt unregistered [ 572.806890] Key type ._llcrypt registered [ 572.809719] Key type .llcrypt registered [ 573.120191] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 573.154472] alg: No test for adler32 (adler32-zlib) [ 574.091644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 574.201477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 574.228723] LNet: 29587:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=16 less than recommended minimum 512 [ 574.239994] LNet: 29587:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 [ 574.255562] LNet: 29587:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 [ 574.293614] LNet: Message forwarding will be enabled in 2s [ 577.379271] LNet: Message forwarding enabled [ 577.899135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 577.943804] LNet: Message forwarding will stop in 2s to 6s [ 581.478400] LNet: Message forwarding disabled [ 582.394475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 582.427313] LNet: 29685:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=16 less than recommended minimum 512 [ 582.434737] LNet: 29685:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 [ 582.448480] LNet: 29685:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 [ 582.473937] LNet: Message forwarding will be enabled in 2s [ 582.621725] LNet: 29698:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 583.218953] Key type .llcrypt unregistered [ 583.225079] Key type ._llcrypt unregistered [ 593.288172] Key type ._llcrypt registered [ 593.290140] Key type .llcrypt registered [ 594.473613] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 15:23:32 (1789673012) [ 594.867764] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 594.890375] alg: No test for adler32 (adler32-zlib) [ 595.759425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 595.821522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 600.912771] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 15:23:39 (1789673019) [ 601.196286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 601.228420] LNet: 31097:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 601.251754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 601.296844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 601.329873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 601.361340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 601.382640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 601.409320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 601.439428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 601.460450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 601.488448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 601.512714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 601.542977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 601.579266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 601.620236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 601.647421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 601.679907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 601.702706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 601.742950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 601.771178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 601.799231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 601.829701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 601.854338] LNetError: 31140:0:(router.c:625:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 607.067376] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 15:23:45 (1789673025) [ 607.336241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 607.350737] LNet: 31711:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 607.362445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 607.400192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 613.727748] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 15:23:51 (1789673031) [ 615.636036] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg658-server [ 617.488902] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 15:23:55 (1789673035) [ 617.645912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 617.704147] LNet: 32640:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 617.739071] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 617.795986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 617.826710] LNet: Added LNI 192.168.206.58@tcp [8/256/0/180] [ 618.659479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.206.[59-63]@tcp [ 618.694483] LNet: 32774:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 618.752162] LNetError: 32645:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.206.59@tcp is being used as a gateway but routing feature is not turned on [ 618.772914] LNetError: 32645:0:(lib-lnet.h:1281:lnet_set_route_aliveness()) route to tcp1 through 192.168.206.59@tcp has gone from up to down [ 621.796745] LNetError: 32775:0:(lib-lnet.h:1281:lnet_set_route_aliveness()) route to tcp1 through 192.168.206.62@tcp has gone from up to down [ 625.543951] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 15:24:03 (1789673043) [ 627.335638] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg658-server [ 629.436410] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 15:24:07 (1789673047) [ 629.536957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 629.565760] LNet: 33703:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 629.583710] LNetError: 33703:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 629.602350] LNet: Removed LNI 192.168.206.58@tcp [ 629.668822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 629.779961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 629.811118] LNet: Added LNI 192.168.206.58@tcp [8/256/0/180] [ 630.574600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.206.[59-64/2]@tcp [ 630.611185] LNet: 33827:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 630.628752] LNet: 33827:0:(router.c:729:lnet_add_route()) Skipped 4 previous similar messages [ 630.665248] LNetError: 33714:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.206.59@tcp is being used as a gateway but routing feature is not turned on [ 630.685509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.206.[59-64/2]@tcp [ 630.691965] LNetError: 33714:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 1 previous similar message [ 630.691990] LNetError: 33714:0:(lib-lnet.h:1281:lnet_set_route_aliveness()) route to tcp103 through 192.168.206.59@tcp has gone from up to down [ 630.691994] LNetError: 33714:0:(lib-lnet.h:1281:lnet_set_route_aliveness()) Skipped 3 previous similar messages [ 637.726300] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 15:24:15 (1789673055) [ 637.843542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 637.874555] LNet: 34414:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 637.884386] LNetError: 34414:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 637.910336] LNet: Removed LNI 192.168.206.58@tcp [ 637.966332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 638.042918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 638.104296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 638.147842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 638.206628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 638.267415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 638.302351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 638.370669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 638.446422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 638.477528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 638.568281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 638.640316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 638.681992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 638.738382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 638.822473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 638.856483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 638.930616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 639.020311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 639.079995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 639.125715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 639.200754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 639.229941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 639.277576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 639.349393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 639.388196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 639.465630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 639.589757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 639.640140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 639.729980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 639.814049] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 639.857081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 639.946393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 640.033463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 640.085186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 640.151829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 646.227250] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 15:24:24 (1789673064) [ 647.927764] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg658-server [ 649.331352] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 15:24:27 (1789673067) [ 650.835336] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg658-server [ 652.402145] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 15:24:30 (1789673070) [ 652.518762] LNet: 35814:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 652.526438] LNet: 35814:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 11 previous similar messages [ 653.185218] Key type .llcrypt unregistered [ 653.187190] Key type ._llcrypt unregistered [ 658.044330] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 658.283048] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 658.294112] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 658.313900] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 660.904738] Key type ._llcrypt registered [ 660.908242] Key type .llcrypt registered [ 661.193177] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 661.235624] alg: No test for adler32 (adler32-zlib) [ 662.273891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 662.332246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 662.362048] LNet: Added LNI 192.168.206.58@tcp [8/256/0/180] [ 663.252223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 663.277200] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 665.794053] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 665.837589] LNet: Removed LNI 10.1.2.3@tcp [ 666.020629] LNet: 36539:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 666.028613] LNetError: 36539:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 666.050481] LNet: Removed LNI 192.168.206.58@tcp [ 666.807749] Key type .llcrypt unregistered [ 666.813994] Key type ._llcrypt unregistered [ 677.715234] Key type ._llcrypt registered [ 677.716753] Key type .llcrypt registered [ 678.724628] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 15:24:57 (1789673097) [ 679.160370] Key type .llcrypt unregistered [ 679.164241] Key type ._llcrypt unregistered [ 684.966412] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 685.187755] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 685.196966] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 685.204167] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 687.704314] Key type ._llcrypt registered [ 687.707790] Key type .llcrypt registered [ 687.882740] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 687.892846] alg: No test for adler32 (adler32-zlib) [ 688.787820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 688.838833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 688.857165] LNet: Added LNI 192.168.206.58@tcp [8/256/0/180] [ 689.523562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 689.539668] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 692.167203] LNet: 38028:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 692.172464] LNetError: 38028:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 692.176621] LNet: Removed LNI 10.1.2.3@tcp [ 692.189878] LNet: Removed LNI 192.168.206.58@tcp [ 692.871411] Key type .llcrypt unregistered [ 692.876141] Key type ._llcrypt unregistered [ 702.198689] Key type ._llcrypt registered [ 702.200626] Key type .llcrypt registered [ 703.048742] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 15:25:21 (1789673121) [ 703.670538] Key type .llcrypt unregistered [ 703.676241] Key type ._llcrypt unregistered [ 708.580104] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 708.702342] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 708.705377] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 708.709377] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 711.057818] Key type ._llcrypt registered [ 711.059820] Key type .llcrypt registered [ 711.206666] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 711.220771] alg: No test for adler32 (adler32-zlib) [ 712.101810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 712.154452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 712.180496] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 713.723194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 713.739199] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 715.489271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 715.515769] LNet: Removed LNI 10.1.2.3@tcp [ 715.539281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 715.584996] LNet: Removed LNI 10.1.2.31@tcp [ 715.603918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 715.637196] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 716.357468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 716.374097] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 717.056584] LNet: 39940:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 717.062421] LNetError: 39940:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg:0 not found [ 717.066944] LNet: Removed LNI 10.1.2.31@tcp [ 717.077189] LNet: Removed LNI 10.1.2.3@tcp [ 717.582743] Key type .llcrypt unregistered [ 717.585678] Key type ._llcrypt unregistered [ 729.565825] Key type ._llcrypt registered [ 729.569290] Key type .llcrypt registered [ 730.929472] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 15:25:48 (1789673148) [ 731.643639] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 731.668086] alg: No test for adler32 (adler32-zlib) [ 732.651584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 732.752655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 732.819724] LNet: Added LNI 192.168.206.58@tcp [8/256/0/180] [ 733.846523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 733.878517] LNet: 40892:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 733.896861] LNetError: 40892:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 733.918986] LNet: Removed LNI 192.168.206.58@tcp [ 733.958970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 734.026305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.206.58@tcp [ 734.102239] LNet: Added LNI 192.168.206.58@tcp [8/256/0/180] [ 735.020572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.206.58@tcp [ 735.055375] LNet: Removed LNI 192.168.206.58@tcp [ 735.128909] LNet: 41027:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 735.720369] Key type .llcrypt unregistered [ 735.723394] Key type ._llcrypt unregistered [ 747.260192] Key type ._llcrypt registered [ 747.265171] Key type .llcrypt registered [ 748.484884] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 15:26:06 (1789673166) [ 750.285468] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg658-server [ 751.829374] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 15:26:10 (1789673170) [ 752.316437] Key type .llcrypt unregistered [ 752.323335] Key type ._llcrypt unregistered [ 757.079131] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 757.241886] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 757.245275] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 757.251687] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 759.547576] Key type ._llcrypt registered [ 759.549353] Key type .llcrypt registered [ 759.657631] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 759.665964] alg: No test for adler32 (adler32-zlib) [ 760.573172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 760.673442] LNet: Added LNI 192.168.206.58@tcp1 [8/256/0/180] [ 761.247472] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 763.289831] LNet: 42858:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 763.297940] LNetError: 42858:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 763.304761] LNet: Removed LNI 192.168.206.58@tcp1 [ 763.322559] LNet: Removed LNI 10.1.2.3@tcp2 [ 763.912473] Key type .llcrypt unregistered [ 763.915052] Key type ._llcrypt unregistered [ 776.737820] Key type ._llcrypt registered [ 776.740287] Key type .llcrypt registered [ 778.352574] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 15:26:35 (1789673195) [ 778.807313] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 778.829774] alg: No test for adler32 (adler32-zlib) [ 779.930666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 780.076872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 780.145177] LNet: Added LNI 192.168.206.58@tcp [8/256/0/180] [ 781.299175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 781.328422] LNet: 43825:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 781.341328] LNetError: 43825:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 781.353660] LNet: Removed LNI 192.168.206.58@tcp [ 781.375509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 781.464314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 781.507824] LNet: Added LNI 192.168.206.58@tcp [8/256/0/180] [ 789.917278] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 15:26:47 (1789673207) [ 789.960324] LNet: 44407:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 789.965601] LNetError: 44407:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 789.976916] LNet: Removed LNI 192.168.206.58@tcp [ 790.627674] Key type .llcrypt unregistered [ 790.629505] Key type ._llcrypt unregistered [ 795.538904] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 795.680657] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 795.683222] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 795.686090] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 798.083847] Key type ._llcrypt registered [ 798.086800] Key type .llcrypt registered [ 798.289587] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 798.304715] alg: No test for adler32 (adler32-zlib) [ 799.228437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 799.287237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 799.310662] LNet: Added LNI 192.168.206.58@tcp [8/256/0/180] [ 799.969734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 799.990582] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 802.077262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 802.104330] LNet: 45152:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 802.109500] LNetError: 45152:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 802.114714] LNet: Removed LNI 10.1.2.3@tcp [ 802.144892] LNet: Removed LNI 192.168.206.58@tcp [ 802.168888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 802.240336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 802.271741] LNet: Added LNI 192.168.206.58@tcp [8/256/0/180] [ 802.277511] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 802.794234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 802.821353] LNet: 45179:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 802.829138] LNetError: 45179:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 802.839577] LNetError: 45179:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 802.846978] LNet: Removed LNI 10.1.2.3@tcp [ 802.871382] LNet: Removed LNI 192.168.206.58@tcp [ 802.910440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 802.970676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 802.990157] LNet: Added LNI 192.168.206.58@tcp [8/256/0/180] [ 802.993642] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 803.382351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 803.401588] LNet: Added LNI 192.168.206.58@tcp2 [8/256/0/180] [ 804.379257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 804.399249] LNet: 45341:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 804.413866] LNetError: 45341:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 804.425306] LNetError: 45341:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 804.442302] LNet: Removed LNI 10.1.2.3@tcp [ 804.444589] LNet: Removed LNI 192.168.206.58@tcp [ 804.461315] LNet: Removed LNI 192.168.206.58@tcp2 [ 804.510303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 804.599813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 804.626793] LNet: Added LNI 192.168.206.58@tcp [8/256/0/180] [ 804.633423] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 804.640341] LNet: Added LNI 192.168.206.58@tcp2 [8/256/0/180] [ 811.571215] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 15:27:09 (1789673229) [ 811.828711] LNet: 45938:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 811.839087] LNetError: 45938:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 811.848834] LNetError: 45938:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 811.860291] LNet: Removed LNI 10.1.2.3@tcp [ 811.868268] LNet: Removed LNI 192.168.206.58@tcp [ 811.887584] LNet: Removed LNI 192.168.206.58@tcp2 [ 812.564047] Key type .llcrypt unregistered [ 812.565745] Key type ._llcrypt unregistered [ 818.249254] Key type ._llcrypt registered [ 818.252574] Key type .llcrypt registered [ 818.463960] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 818.476202] alg: No test for adler32 (adler32-zlib) [ 819.510485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 819.575182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 819.598228] LNet: Added LNI 192.168.206.58@tcp [8/256/0/180] [ 820.289619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 820.311488] LNet: 46329:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 820.321077] LNetError: 46329:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 820.337818] LNet: Removed LNI 192.168.206.58@tcp [ 820.361503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 820.435755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 820.461631] LNet: Added LNI 192.168.206.58@tcp [8/256/0/180] [ 820.943944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 820.966142] LNet: 46359:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 820.975307] LNetError: 46359:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 820.999912] LNet: Removed LNI 192.168.206.58@tcp [ 821.030676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 821.078756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 821.098655] LNet: Added LNI 192.168.206.58@tcp [8/256/0/180] [ 821.446900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 821.470163] LNet: Removed LNI 192.168.206.58@tcp [ 821.498526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 821.554657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 821.584460] LNet: Added LNI 192.168.206.58@tcp [8/256/0/180] [ 821.870470] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 822.023023] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 822.026682] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 822.039667] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 824.353220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 824.370249] LNet: 46468:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 824.376794] LNet: 46468:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 824.382938] LNetError: 46468:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 824.387936] LNetError: 46468:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 824.399305] LNet: Removed LNI 192.168.206.58@tcp [ 824.419953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 824.515766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 824.555622] LNet: Added LNI 192.168.206.58@tcp [8/256/0/180] [ 830.807948] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 15:27:28 (1789673248) [ 830.932203] LNet: 47061:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 830.951351] LNetError: 47061:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 830.976549] LNet: Removed LNI 192.168.206.58@tcp [ 831.527943] Key type .llcrypt unregistered [ 831.531625] Key type ._llcrypt unregistered [ 838.491059] Key type ._llcrypt registered [ 838.493419] Key type .llcrypt registered [ 838.698698] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 838.715206] alg: No test for adler32 (adler32-zlib) [ 839.622775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 839.695606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 839.731054] LNet: Added LNI 192.168.206.58@tcp [8/256/0/180] [ 840.519432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 840.536376] LNet: 47461:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 840.556952] LNetError: 47461:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 840.579187] LNet: Removed LNI 192.168.206.58@tcp [ 840.600509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 840.652601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 840.683570] LNet: Added LNI 192.168.206.58@tcp [8/256/0/180] [ 847.427389] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 15:27:45 (1789673265) [ 847.535187] LNet: 48046:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 847.540728] LNetError: 48046:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 847.562505] LNet: Removed LNI 192.168.206.58@tcp [ 848.097476] Key type .llcrypt unregistered [ 848.105959] Key type ._llcrypt unregistered [ 853.334302] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 853.507407] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 853.512123] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 853.518194] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 854.874955] Key type ._llcrypt registered [ 854.879104] Key type .llcrypt registered [ 855.054118] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 855.075734] alg: No test for adler32 (adler32-zlib) [ 856.062749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 856.109905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 856.127857] LNet: Added LNI 192.168.206.58@tcp [8/256/0/180] [ 857.008916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 857.025105] LNet: 48493:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 857.030600] LNetError: 48493:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 857.046901] LNet: Removed LNI 192.168.206.58@tcp [ 857.065690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 857.105758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 857.138712] LNet: Added LNI 192.168.206.58@tcp [8/256/0/180] [ 863.635406] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 15:28:01 (1789673281) [ 863.777064] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 863.813068] LNet: 49085:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 863.826473] LNetError: 49085:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 863.849966] LNet: Removed LNI 192.168.206.58@tcp [ 863.888963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 864.004770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 869.399825] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 15:28:07 (1789673287) [ 869.506430] LNet: 49667:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 870.061901] Key type .llcrypt unregistered [ 870.065258] Key type ._llcrypt unregistered [ 875.255410] Key type ._llcrypt registered [ 875.258234] Key type .llcrypt registered [ 875.395255] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 875.405501] alg: No test for adler32 (adler32-zlib) [ 876.302149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 876.343661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 876.372176] LNet: Added LNI 192.168.206.58@tcp [8/256/0/180] [ 877.190926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 877.213328] LNet: 50068:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 877.221685] LNetError: 50068:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 877.242842] LNet: Removed LNI 192.168.206.58@tcp [ 877.271698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 877.371630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 877.407801] LNet: Added LNI 192.168.206.58@tcp [8/256/0/180] [ 883.331024] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 15:28:21 (1789673301) [ 883.489606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 883.528599] LNet: 50661:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 883.540677] LNetError: 50661:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 883.557842] LNet: Removed LNI 192.168.206.58@tcp [ 883.606108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 883.664720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 889.486229] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 15:28:27 (1789673307) [ 890.636512] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 892.455913] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 15:28:30 (1789673310) [ 894.199541] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 896.120908] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 15:28:33 (1789673313) [ 897.910853] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 900.622720] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 15:28:37 (1789673317) [ 902.575895] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 904.796734] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 15:28:42 (1789673322) [ 906.668363] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 908.077995] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing ========================================================== 15:28:46 (1789673326) [ 908.483065] LNet: 52685:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 909.280751] Key type .llcrypt unregistered [ 909.283527] Key type ._llcrypt unregistered [ 916.818409] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 917.031484] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 917.034614] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 917.037872] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 919.570395] Key type ._llcrypt registered [ 919.572747] Key type .llcrypt registered [ 919.743534] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 919.753667] alg: No test for adler32 (adler32-zlib) [ 920.707374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 920.794363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 920.829635] LNet: Added LNI 192.168.206.58@tcp [8/512/0/180] [ 920.839240] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 920.846880] LNet: 53004:0:(api-ni.c:2991:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 920.952538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 920.992711] LNet: 53023:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 921.001980] LNetError: 53023:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 921.007681] LNet: Removed LNI 10.1.2.3@tcp [ 921.014399] LNetError: 53023:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 921.032614] LNet: Removed LNI 192.168.206.58@tcp [ 921.055579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 921.128123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 921.162393] LNet: Added LNI 192.168.206.58@tcp [8/256/0/180] [ 921.220472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 921.244210] LNet: 53047:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 921.257046] LNetError: 53047:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 921.267424] LNet: Removed LNI 192.168.206.58@tcp [ 921.284394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 921.338403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 921.377374] LNet: Added LNI 192.168.206.58@tcp [8/256/0/180] [ 921.384853] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 921.390969] LNet: 53055:0:(api-ni.c:2991:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 921.519429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 921.537045] LNet: 53073:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 921.541581] LNetError: 53073:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 921.545946] LNet: Removed LNI 10.1.2.3@tcp [ 921.547696] LNetError: 53073:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 921.560085] LNet: Removed LNI 192.168.206.58@tcp [ 921.576209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 921.618594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 921.637853] LNet: Added LNI 192.168.206.58@tcp [8/256/0/180] [ 921.641843] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 921.645676] LNet: 53081:0:(api-ni.c:2991:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 921.866661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 921.885678] LNet: 53107:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 921.890896] LNetError: 53107:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 921.894884] LNet: Removed LNI 10.1.2.3@tcp [ 921.898682] LNetError: 53107:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 921.909946] LNet: Removed LNI 192.168.206.58@tcp [ 921.943494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 922.008424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 922.053251] LNet: Added LNI 192.168.206.58@tcp [8/256/0/182] [ 922.059701] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 922.185316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 922.212895] LNet: 53136:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 922.219538] LNetError: 53136:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 922.229043] LNet: Removed LNI 10.1.2.3@tcp [ 922.238347] LNetError: 53136:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 922.261853] LNet: Removed LNI 192.168.206.58@tcp [ 922.288889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 922.350979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 922.367865] LNet: Added LNI 192.168.206.58@tcp [8/256/0/182] [ 922.377589] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 922.380332] LNet: 53144:0:(api-ni.c:2991:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 922.448224] LNet: Removed LNI 10.1.2.3@tcp [ 922.460394] LNet: Removed LNI 192.168.206.58@tcp [ 923.101057] Key type .llcrypt unregistered [ 923.105069] Key type ._llcrypt unregistered [ 934.059431] Key type ._llcrypt registered [ 934.061810] Key type .llcrypt registered [ 935.380502] Lustre: DEBUG MARKER: == sanity-lnet test 180: FastReg map error should not leak descriptors ========================================================== 15:29:13 (1789673353) [ 936.700609] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_180 Need o2ib NETTYPE [ 938.657957] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 15:29:16 (1789673356) [ 939.017950] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 939.030533] alg: No test for adler32 (adler32-zlib) [ 939.970141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 940.023085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 940.062618] LNet: Added LNI 192.168.206.58@tcp [8/256/0/180] [ 941.029127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.206.58 [ 941.061521] LNet: 54405:0:(config.c:1740:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 941.068528] LNet: Removed LNI 192.168.206.58@tcp [ 947.357297] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 15:29:25 (1789673365) [ 947.434317] LNet: 54966:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 947.443959] LNetError: 54966:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 947.458053] LNet: Removed LNI 192.168.206.58@tcp [ 948.073427] Key type .llcrypt unregistered [ 948.077370] Key type ._llcrypt unregistered [ 954.430806] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 954.715260] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 957.379515] Key type ._llcrypt registered [ 957.382411] Key type .llcrypt registered [ 957.504975] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 957.517570] alg: No test for adler32 (adler32-zlib) [ 958.791970] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 964.364698] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 15:29:42 (1789673382) [ 964.605153] LNet: 55885:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 964.612549] LNetError: 55885:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 964.633243] LNet: Removed LNI 10.1.2.3@tcp [ 965.204469] Key type .llcrypt unregistered [ 965.210533] Key type ._llcrypt unregistered [ 970.514213] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 970.780086] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 970.783980] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 970.789568] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 972.360138] Key type ._llcrypt registered [ 972.364453] Key type .llcrypt registered [ 972.565188] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 972.585875] alg: No test for adler32 (adler32-zlib) [ 973.549872] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 979.410257] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 15:29:57 (1789673397) [ 980.733784] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 981.982376] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 15:30:00 (1789673400) [ 982.032337] LNet: 57049:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 982.036210] LNetError: 57049:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 982.047648] LNet: Removed LNI 10.1.2.3@tcp [ 982.462985] Key type .llcrypt unregistered [ 982.464587] Key type ._llcrypt unregistered [ 986.673684] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 987.032198] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 987.037679] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 987.041580] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 988.570953] Key type ._llcrypt registered [ 988.576325] Key type .llcrypt registered [ 988.788334] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 988.818846] alg: No test for adler32 (adler32-zlib) [ 989.684695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 989.758681] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 999.405266] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 15:30:17 (1789673417) [ 999.786610] LNet: 58277:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 999.801378] LNetError: 58277:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 999.816902] LNet: Removed LNI 10.1.2.3@tcp [ 1000.773567] Key type .llcrypt unregistered [ 1000.778800] Key type ._llcrypt unregistered [ 1010.037383] Key type ._llcrypt registered [ 1010.039683] Key type .llcrypt registered [ 1010.389649] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1010.416383] alg: No test for adler32 (adler32-zlib) [ 1011.504318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1011.532985] LNet: Added LNI 192.168.206.58@tcp [8/256/0/180] [ 1018.638700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.158@tcp [ 1019.216459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.58@tcp -d 192.168.206.158@tcp -m GET -e local_interrupt -r 1 [ 1019.245600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.58@tcp -d 192.168.206.58@tcp -m GET -e local_interrupt -r 1 [ 1019.271134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.158@tcp [ 1019.285973] LNet: There was an unexpected network error while writing to 192.168.206.158: rc = -22 [ 1019.448596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1019.476528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1019.501926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1019.523587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1019.547665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1019.569839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1019.592573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1019.621278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1019.643615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1019.655636] LNet: 58714:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1019.668313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1019.693125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1019.730807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1019.744414] LNet: 1 local NIs in recovery (showing 1): 192.168.206.58@tcp [ 1019.759160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1019.788927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1019.813275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1019.840255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1020.319070] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.58@tcp -d 192.168.206.158@tcp -m GET -e local_dropped -r 1 [ 1020.353502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.58@tcp -d 192.168.206.58@tcp -m GET -e local_dropped -r 1 [ 1020.379626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.158@tcp [ 1020.396424] LNet: There was an unexpected network error while writing to 192.168.206.158: rc = -22 [ 1020.549583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1020.579919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1020.603891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1020.639845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1020.670280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1020.705266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1020.755323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1020.800791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1020.833482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1020.852503] LNet: 58825:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1020.871148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1020.893582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1020.922865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1020.967587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1021.008156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1021.034676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1021.072212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1021.417874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.58@tcp -d 192.168.206.158@tcp -m GET -e local_aborted -r 1 [ 1021.458103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.58@tcp -d 192.168.206.58@tcp -m GET -e local_aborted -r 1 [ 1021.484344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.158@tcp [ 1021.527443] LNet: There was an unexpected network error while writing to 192.168.206.158: rc = -22 [ 1021.534675] LNet: Skipped 1 previous similar message [ 1021.686300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1021.715711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1021.747660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1021.782530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1021.817942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1021.840974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1021.869326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1021.892884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1021.927962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1021.943785] LNet: 58936:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1021.964253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1022.012156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1022.038307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1022.061506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1022.083842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1022.111354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1022.141083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1022.636605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.58@tcp -d 192.168.206.158@tcp -m GET -e local_no_route -r 1 [ 1022.670661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.58@tcp -d 192.168.206.58@tcp -m GET -e local_no_route -r 1 [ 1022.699610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.158@tcp [ 1022.912569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1022.943675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1022.970987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1023.001850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1023.029164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1023.077030] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1023.133606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1023.176036] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1023.217645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1023.278590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1023.319470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1023.359744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1023.411562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1023.462361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1023.508498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1023.554493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1024.088746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.58@tcp -d 192.168.206.158@tcp -m GET -e local_timeout -r 1 [ 1024.127156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.58@tcp -d 192.168.206.58@tcp -m GET -e local_timeout -r 1 [ 1024.175443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.158@tcp [ 1024.198794] LNet: There was an unexpected network error while writing to 192.168.206.158: rc = -22 [ 1024.204365] LNet: Skipped 2 previous similar messages [ 1024.520280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1024.545300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1024.581438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1024.609389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1024.633965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1024.664081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1024.704825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1024.734966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1024.760289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1024.772278] LNet: 59158:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1024.775592] LNet: 59158:0:(api-ni.c:301:recovery_interval_set()) Skipped 1 previous similar message [ 1024.785659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1024.809671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1024.828478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1024.860377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1024.893134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1024.933791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1024.958398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1025.300392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.58@tcp -d 192.168.206.158@tcp -m GET -e local_error -r 1 [ 1025.337877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.58@tcp -d 192.168.206.58@tcp -m GET -e local_error -r 1 [ 1025.361778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.158@tcp [ 1025.610620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1025.631403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1025.651602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1025.675946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1025.701519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1025.729608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1025.762817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1025.786240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1025.820215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1025.845353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1025.886602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1025.916459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1025.973254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1026.023714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1026.074814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1026.113869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1026.486559] LNet: 59311:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1026.508544] LNetError: 59311:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1026.525778] LNet: Removed LNI 192.168.206.58@tcp [ 1026.971494] Key type .llcrypt unregistered [ 1026.973769] Key type ._llcrypt unregistered [ 1041.524971] Key type ._llcrypt registered [ 1041.527978] Key type .llcrypt registered [ 1042.789408] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 15:31:00 (1789673460) [ 1043.650656] Key type .llcrypt unregistered [ 1043.653277] Key type ._llcrypt unregistered [ 1049.710872] Key type ._llcrypt registered [ 1049.714307] Key type .llcrypt registered [ 1049.940548] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1049.975240] alg: No test for adler32 (adler32-zlib) [ 1051.011350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1051.063954] LNet: Added LNI 192.168.206.58@tcp [8/256/0/180] [ 1058.525660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.158@tcp [ 1069.764782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1069.815514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1069.847174] LNet: Added LNI 192.168.206.58@tcp1 [8/256/0/180] [ 1071.399550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.58@tcp -d 192.168.206.158@tcp -m GET -e local_interrupt -r 1 [ 1071.469659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.58@tcp -d 192.168.206.158@tcp1 -m GET -e local_interrupt -r 1 [ 1071.521703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.58@tcp -d 192.168.206.58@tcp -m GET -e local_interrupt -r 1 [ 1071.593260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.58@tcp -d 192.168.206.58@tcp1 -m GET -e local_interrupt -r 1 [ 1071.633786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.58@tcp1 -d 192.168.206.158@tcp -m GET -e local_interrupt -r 1 [ 1071.706933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.58@tcp1 -d 192.168.206.158@tcp1 -m GET -e local_interrupt -r 1 [ 1071.785621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.58@tcp1 -d 192.168.206.58@tcp -m GET -e local_interrupt -r 1 [ 1071.844642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.58@tcp1 -d 192.168.206.58@tcp1 -m GET -e local_interrupt -r 1 [ 1071.894503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.158@tcp [ 1071.942501] LNet: There was an unexpected network error while writing to 192.168.206.158: rc = -22 [ 1071.951462] LNet: 1 local NIs in recovery (showing 1): 192.168.206.58@tcp [ 1072.220521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1072.250629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1072.293776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1072.325276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1072.366921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1072.396818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1072.444703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1072.475805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1072.506385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1072.527921] LNet: 60774:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1072.542802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1072.565703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1072.588422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1072.607756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1072.633170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1072.656610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1072.684711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1073.241818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.58@tcp -d 192.168.206.158@tcp -m GET -e local_dropped -r 1 [ 1073.275699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.58@tcp -d 192.168.206.158@tcp1 -m GET -e local_dropped -r 1 [ 1073.334623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.58@tcp -d 192.168.206.58@tcp -m GET -e local_dropped -r 1 [ 1073.388412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.58@tcp -d 192.168.206.58@tcp1 -m GET -e local_dropped -r 1 [ 1073.431183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.58@tcp1 -d 192.168.206.158@tcp -m GET -e local_dropped -r 1 [ 1073.485534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.58@tcp1 -d 192.168.206.158@tcp1 -m GET -e local_dropped -r 1 [ 1073.574710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.58@tcp1 -d 192.168.206.58@tcp -m GET -e local_dropped -r 1 [ 1073.651757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.58@tcp1 -d 192.168.206.58@tcp1 -m GET -e local_dropped -r 1 [ 1073.686654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.158@tcp [ 1073.706690] LNet: There was an unexpected network error while writing to 192.168.206.158: rc = -22 [ 1073.716700] LNet: Skipped 2 previous similar messages [ 1074.019879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1074.050379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1074.118887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1074.171324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1074.204436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1074.245677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1074.284910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1074.311790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1074.337418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1074.349686] LNet: 60898:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1074.367346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1074.405339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1074.471451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1074.528660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1074.571733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1074.597512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1074.631413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1075.110657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.58@tcp -d 192.168.206.158@tcp -m GET -e local_aborted -r 1 [ 1075.185663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.58@tcp -d 192.168.206.158@tcp1 -m GET -e local_aborted -r 1 [ 1075.243561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.58@tcp -d 192.168.206.58@tcp -m GET -e local_aborted -r 1 [ 1075.293781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.58@tcp -d 192.168.206.58@tcp1 -m GET -e local_aborted -r 1 [ 1075.352044] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.58@tcp1 -d 192.168.206.158@tcp -m GET -e local_aborted -r 1 [ 1075.422756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.58@tcp1 -d 192.168.206.158@tcp1 -m GET -e local_aborted -r 1 [ 1075.475527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.58@tcp1 -d 192.168.206.58@tcp -m GET -e local_aborted -r 1 [ 1075.529505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.58@tcp1 -d 192.168.206.58@tcp1 -m GET -e local_aborted -r 1 [ 1075.596806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.158@tcp [ 1075.647810] LNet: There was an unexpected network error while writing to 192.168.206.158: rc = -22 [ 1075.662329] LNet: Skipped 2 previous similar messages [ 1075.895704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1075.932588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1075.970356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1075.993275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1076.021875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1076.063096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1076.116657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1076.158987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1076.198051] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1076.227249] LNet: 61021:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1076.256473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1076.282412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1076.328426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1076.363323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1076.417594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1076.471198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1076.506409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1077.007676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.58@tcp -d 192.168.206.158@tcp -m GET -e local_no_route -r 1 [ 1077.032603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.58@tcp -d 192.168.206.158@tcp1 -m GET -e local_no_route -r 1 [ 1077.054593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.58@tcp -d 192.168.206.58@tcp -m GET -e local_no_route -r 1 [ 1077.084935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.58@tcp -d 192.168.206.58@tcp1 -m GET -e local_no_route -r 1 [ 1077.130227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.58@tcp1 -d 192.168.206.158@tcp -m GET -e local_no_route -r 1 [ 1077.167870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.58@tcp1 -d 192.168.206.158@tcp1 -m GET -e local_no_route -r 1 [ 1077.199318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.58@tcp1 -d 192.168.206.58@tcp -m GET -e local_no_route -r 1 [ 1077.231627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.58@tcp1 -d 192.168.206.58@tcp1 -m GET -e local_no_route -r 1 [ 1077.265370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.158@tcp [ 1077.504313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1077.535562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1077.562789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1077.588860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1077.616862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1077.654138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1077.683117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1077.737055] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1077.793567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1077.842410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1077.896957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1077.937295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1078.006877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1078.072624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1078.124597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1078.179890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1078.636546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.58@tcp -d 192.168.206.158@tcp -m GET -e local_timeout -r 1 [ 1078.660832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.58@tcp -d 192.168.206.158@tcp1 -m GET -e local_timeout -r 1 [ 1078.689577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.58@tcp -d 192.168.206.58@tcp -m GET -e local_timeout -r 1 [ 1078.719404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.58@tcp -d 192.168.206.58@tcp1 -m GET -e local_timeout -r 1 [ 1078.750909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.58@tcp1 -d 192.168.206.158@tcp -m GET -e local_timeout -r 1 [ 1078.808330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.58@tcp1 -d 192.168.206.158@tcp1 -m GET -e local_timeout -r 1 [ 1078.852210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.58@tcp1 -d 192.168.206.58@tcp -m GET -e local_timeout -r 1 [ 1078.900689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.58@tcp1 -d 192.168.206.58@tcp1 -m GET -e local_timeout -r 1 [ 1078.948636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.158@tcp [ 1078.963961] LNet: There was an unexpected network error while writing to 192.168.206.158: rc = -22 [ 1078.967923] LNet: Skipped 5 previous similar messages [ 1079.166626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1079.189928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1079.242095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1079.322284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1079.349648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1079.385989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1079.433106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1079.471184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1079.509839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1079.539918] LNet: 61267:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1079.544820] LNet: 61267:0:(api-ni.c:301:recovery_interval_set()) Skipped 1 previous similar message [ 1079.573617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1079.615848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1079.648293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1079.677557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1079.706190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1079.756403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1079.806535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1080.247554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.58@tcp -d 192.168.206.158@tcp -m GET -e local_error -r 1 [ 1080.270273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.58@tcp -d 192.168.206.158@tcp1 -m GET -e local_error -r 1 [ 1080.295267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.58@tcp -d 192.168.206.58@tcp -m GET -e local_error -r 1 [ 1080.327863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.58@tcp -d 192.168.206.58@tcp1 -m GET -e local_error -r 1 [ 1080.375858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.58@tcp1 -d 192.168.206.158@tcp -m GET -e local_error -r 1 [ 1080.409929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.58@tcp1 -d 192.168.206.158@tcp1 -m GET -e local_error -r 1 [ 1080.452288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.58@tcp1 -d 192.168.206.58@tcp -m GET -e local_error -r 1 [ 1080.480840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.58@tcp1 -d 192.168.206.58@tcp1 -m GET -e local_error -r 1 [ 1080.506421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.158@tcp [ 1080.664408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1080.684442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1080.719861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1080.761081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1080.791406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1080.821180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1080.856381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1080.894567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1080.963864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1081.033076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1081.076812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1081.136532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1081.176129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1081.204894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1081.244165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1081.280060] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1086.745214] LNet: 61466:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1086.750874] LNetError: 61466:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1087.777715] LNet: Removed LNI 192.168.206.58@tcp [ 1087.780350] LNetError: 61466:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1087.822536] LNet: Removed LNI 192.168.206.58@tcp1 [ 1088.470575] Key type .llcrypt unregistered [ 1088.473415] Key type ._llcrypt unregistered [ 1106.811940] Key type ._llcrypt registered [ 1106.815487] Key type .llcrypt registered [ 1108.111986] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 15:32:06 (1789673526) [ 1109.136361] Key type .llcrypt unregistered [ 1109.138431] Key type ._llcrypt unregistered [ 1115.943837] Key type ._llcrypt registered [ 1115.947975] Key type .llcrypt registered [ 1116.179757] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1116.211937] alg: No test for adler32 (adler32-zlib) [ 1117.137494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1117.167156] LNet: Added LNI 192.168.206.58@tcp [8/256/0/180] [ 1125.582491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.158@tcp [ 1126.286403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.58@tcp -d 192.168.206.158@tcp -m GET -e remote_dropped -r 1 [ 1126.320666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.58@tcp -d 192.168.206.58@tcp -m GET -e remote_dropped -r 1 [ 1126.378472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.158@tcp [ 1126.398911] LNet: There was an unexpected network error while writing to 192.168.206.158: rc = -22 [ 1126.656557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1126.699925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1126.746824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1126.779866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1126.816804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1126.877463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1126.912682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1126.962649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1127.016490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1127.055726] LNet: 62714:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1127.080538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1127.128566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1127.200674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1127.250281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1127.288651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1127.345613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1127.387501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1127.951143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.58@tcp -d 192.168.206.158@tcp -m GET -e remote_error -r 1 [ 1127.989934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.58@tcp -d 192.168.206.58@tcp -m GET -e remote_error -r 1 [ 1128.030272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.158@tcp [ 1128.060697] LNet: There was an unexpected network error while writing to 192.168.206.158: rc = -22 [ 1128.319658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1128.361226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1128.414799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1128.466346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1128.542239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1128.602899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1128.666751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1128.709353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1128.742826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1128.769656] LNet: 62825:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1128.798615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1128.825499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1128.862885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1128.911232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1128.947558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1128.993230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1129.036383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1129.568644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.58@tcp -d 192.168.206.158@tcp -m GET -e remote_timeout -r 1 [ 1129.625130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.58@tcp -d 192.168.206.58@tcp -m GET -e remote_timeout -r 1 [ 1129.675386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.158@tcp [ 1129.702793] LNet: There was an unexpected network error while writing to 192.168.206.158: rc = -22 [ 1129.941840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1129.979076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1130.019979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1130.056454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1130.083719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1130.113544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1130.138743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1130.161440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1130.190309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1130.204644] LNet: 62936:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1130.220961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1130.245728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1130.276271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1130.306897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1130.334983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1130.367371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1130.393866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1130.671854] LNet: 62977:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1130.680261] LNetError: 62977:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1131.751077] LNet: Removed LNI 192.168.206.58@tcp [ 1132.174727] Key type .llcrypt unregistered [ 1132.180950] Key type ._llcrypt unregistered [ 1148.549582] Key type ._llcrypt registered [ 1148.556599] Key type .llcrypt registered [ 1149.878300] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 15:32:47 (1789673567) [ 1150.760414] Key type .llcrypt unregistered [ 1150.766429] Key type ._llcrypt unregistered [ 1156.445419] Key type ._llcrypt registered [ 1156.447738] Key type .llcrypt registered [ 1156.589322] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1156.605835] alg: No test for adler32 (adler32-zlib) [ 1157.492537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1157.528843] LNet: Added LNI 192.168.206.58@tcp [8/256/0/180] [ 1164.261359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.158@tcp [ 1173.144711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1173.197620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1173.235752] LNet: Added LNI 192.168.206.58@tcp1 [8/256/0/180] [ 1174.932186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.58@tcp -d 192.168.206.158@tcp -m GET -e remote_dropped -r 1 [ 1174.962957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.58@tcp -d 192.168.206.158@tcp1 -m GET -e remote_dropped -r 1 [ 1174.988295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.58@tcp -d 192.168.206.58@tcp -m GET -e remote_dropped -r 1 [ 1175.011743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.58@tcp -d 192.168.206.58@tcp1 -m GET -e remote_dropped -r 1 [ 1175.035594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.58@tcp1 -d 192.168.206.158@tcp -m GET -e remote_dropped -r 1 [ 1175.063584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.58@tcp1 -d 192.168.206.158@tcp1 -m GET -e remote_dropped -r 1 [ 1175.086799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.58@tcp1 -d 192.168.206.58@tcp -m GET -e remote_dropped -r 1 [ 1175.111775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.58@tcp1 -d 192.168.206.58@tcp1 -m GET -e remote_dropped -r 1 [ 1175.150830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.158@tcp [ 1175.194311] LNet: There was an unexpected network error while writing to 192.168.206.158: rc = -22 [ 1175.202361] LNet: 1 peer NIs in recovery (showing 1): 192.168.206.158@tcp [ 1175.437648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1175.465263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1175.530880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1175.588508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1175.644804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1175.691813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1175.741708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1175.777598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1175.810478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1175.828587] LNet: 64440:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1175.849958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1175.911562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1175.957810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1176.000681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1176.051765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1176.107891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1176.170867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1176.216126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1176.711103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.58@tcp -d 192.168.206.158@tcp -m GET -e remote_error -r 1 [ 1176.746800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.58@tcp -d 192.168.206.158@tcp1 -m GET -e remote_error -r 1 [ 1176.788417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.58@tcp -d 192.168.206.58@tcp -m GET -e remote_error -r 1 [ 1176.828553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.58@tcp -d 192.168.206.58@tcp1 -m GET -e remote_error -r 1 [ 1176.897321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.58@tcp1 -d 192.168.206.158@tcp -m GET -e remote_error -r 1 [ 1176.953451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.58@tcp1 -d 192.168.206.158@tcp1 -m GET -e remote_error -r 1 [ 1176.992568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.58@tcp1 -d 192.168.206.58@tcp -m GET -e remote_error -r 1 [ 1177.043739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.58@tcp1 -d 192.168.206.58@tcp1 -m GET -e remote_error -r 1 [ 1177.087797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.158@tcp [ 1177.111628] LNet: There was an unexpected network error while writing to 192.168.206.158: rc = -22 [ 1177.119556] LNet: Skipped 2 previous similar messages [ 1177.366583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1177.400687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1177.432490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1177.487609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1177.542945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1177.607829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1177.652189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1177.692867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1177.723676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1177.740543] LNet: 64566:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1177.763232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1177.803408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1177.858420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1177.913396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1177.951414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1177.986664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1178.031725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1178.064147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1178.526087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.58@tcp -d 192.168.206.158@tcp -m GET -e remote_timeout -r 1 [ 1178.557768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.58@tcp -d 192.168.206.158@tcp1 -m GET -e remote_timeout -r 1 [ 1178.596550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.58@tcp -d 192.168.206.58@tcp -m GET -e remote_timeout -r 1 [ 1178.629493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.58@tcp -d 192.168.206.58@tcp1 -m GET -e remote_timeout -r 1 [ 1178.660045] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.58@tcp1 -d 192.168.206.158@tcp -m GET -e remote_timeout -r 1 [ 1178.688608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.58@tcp1 -d 192.168.206.158@tcp1 -m GET -e remote_timeout -r 1 [ 1178.716805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.58@tcp1 -d 192.168.206.58@tcp -m GET -e remote_timeout -r 1 [ 1178.756251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.58@tcp1 -d 192.168.206.58@tcp1 -m GET -e remote_timeout -r 1 [ 1178.798261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.158@tcp [ 1178.823209] LNet: There was an unexpected network error while writing to 192.168.206.158: rc = -22 [ 1179.033271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1179.056513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1179.080138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1179.110344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1179.134281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1179.158644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1179.185769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1179.215688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1179.239423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1179.255543] LNet: 64691:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1179.272506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1179.297297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1179.318257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1179.343530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1179.376574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1179.412760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1179.438893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1179.489627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1183.909441] LNet: 64769:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1183.915662] LNetError: 64769:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1184.928268] LNet: Removed LNI 192.168.206.58@tcp [ 1184.930375] LNetError: 64769:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1184.947782] LNet: Removed LNI 192.168.206.58@tcp1 [ 1185.395153] Key type .llcrypt unregistered [ 1185.400250] Key type ._llcrypt unregistered [ 1201.285481] Key type ._llcrypt registered [ 1201.288412] Key type .llcrypt registered [ 1202.492467] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 15:33:40 (1789673620) [ 1204.022172] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 1205.754200] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 15:33:43 (1789673623) [ 1206.524746] Key type .llcrypt unregistered [ 1206.528826] Key type ._llcrypt unregistered [ 1211.567506] Key type ._llcrypt registered [ 1211.570319] Key type .llcrypt registered [ 1211.737259] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1211.744193] alg: No test for adler32 (adler32-zlib) [ 1212.638808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1212.669617] LNet: Added LNI 192.168.206.58@tcp [8/256/0/180] [ 1219.165178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.158@tcp [ 1220.141785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.58@tcp -d 192.168.206.158@tcp -m GET -e network_timeout -r 1 [ 1220.184463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.58@tcp -d 192.168.206.58@tcp -m GET -e network_timeout -r 1 [ 1220.249938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.158@tcp [ 1220.277758] LNet: There was an unexpected network error while writing to 192.168.206.158: rc = -22 [ 1220.465469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1220.493148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1220.513942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1220.548385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1220.583705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1220.625416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1220.673595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1220.721926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1220.761799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1220.789326] LNet: 66308:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1220.815630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1220.859674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1220.896491] LNet: 1 local NIs in recovery (showing 1): 192.168.206.58@tcp [ 1220.908568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1220.973150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1221.026820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1221.083853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1221.413671] LNet: 66347:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1221.420866] LNetError: 66347:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1221.438601] LNet: Removed LNI 192.168.206.58@tcp [ 1221.849734] Key type .llcrypt unregistered [ 1221.854949] Key type ._llcrypt unregistered [ 1235.630189] Key type ._llcrypt registered [ 1235.633304] Key type .llcrypt registered [ 1235.815511] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1235.827536] alg: No test for adler32 (adler32-zlib) [ 1236.666653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1236.684962] LNet: Added LNI 192.168.206.58@tcp [8/256/0/180] [ 1246.961144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.158@tcp [ 1258.787426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1258.843153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1258.883042] LNet: Added LNI 192.168.206.58@tcp1 [8/256/0/180] [ 1260.335194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.58@tcp -d 192.168.206.158@tcp -m GET -e network_timeout -r 1 [ 1260.406795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.58@tcp -d 192.168.206.158@tcp1 -m GET -e network_timeout -r 1 [ 1260.489175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.58@tcp -d 192.168.206.58@tcp -m GET -e network_timeout -r 1 [ 1260.566464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.58@tcp -d 192.168.206.58@tcp1 -m GET -e network_timeout -r 1 [ 1260.643359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.58@tcp1 -d 192.168.206.158@tcp -m GET -e network_timeout -r 1 [ 1260.722693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.58@tcp1 -d 192.168.206.158@tcp1 -m GET -e network_timeout -r 1 [ 1260.793343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.58@tcp1 -d 192.168.206.58@tcp -m GET -e network_timeout -r 1 [ 1260.865249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.58@tcp1 -d 192.168.206.58@tcp1 -m GET -e network_timeout -r 1 [ 1260.920932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.158@tcp [ 1260.942980] LNet: There was an unexpected network error while writing to 192.168.206.158: rc = -22 [ 1261.293891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1261.344354] LNet: 1 local NIs in recovery (showing 1): 192.168.206.58@tcp [ 1261.349992] LNet: 1 peer NIs in recovery (showing 1): 192.168.206.158@tcp [ 1261.392667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1261.466777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1261.510083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1261.537777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1261.556915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1261.578639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1261.605611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1261.630264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1261.642303] LNet: 67223:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1261.653684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1261.684543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1261.713289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1261.741645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1261.763872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1261.792207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1265.158417] LNet: There was an unexpected network error while writing to 192.168.206.158: rc = -22 [ 1265.169449] LNet: Skipped 2 previous similar messages [ 1265.479132] LNet: 67297:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1265.483379] LNetError: 67297:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1265.490549] LNet: Removed LNI 192.168.206.58@tcp [ 1265.500358] LNet: Removed LNI 192.168.206.58@tcp1 [ 1265.836447] Key type .llcrypt unregistered [ 1265.838352] Key type ._llcrypt unregistered [ 1278.937912] Key type ._llcrypt registered [ 1278.940592] Key type .llcrypt registered [ 1280.174050] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 15:34:58 (1789673698) [ 1280.638684] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1280.663549] alg: No test for adler32 (adler32-zlib) [ 1281.580678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1281.658048] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1281.686439] LNet: Added LNI 192.168.206.58@tcp [8/256/0/180] [ 1282.550715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1282.570539] LNet: Added LNI 192.168.206.58@tcp1 [8/256/0/180] [ 1283.568950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.58@tcp [ 1283.706228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1283.780698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1283.825888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1283.897146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1283.955804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1283.980608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.206.58@tcp [ 1284.023208] LNetError: 68394:0:(api-ni.c:4478:lnet_ni_set_healthv()) manually adding local NI 192.168.206.58@tcp to recovery [ 1284.704576] LNet: There was an unexpected network error while writing to 192.168.206.58: rc = -22 [ 1284.704427] LNet: 1 local NIs in recovery (showing 1): 192.168.206.58@tcp [ 1286.752472] LNet: There was an unexpected network error while writing to 192.168.206.58: rc = -22 [ 1287.684615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1290.848463] LNet: There was an unexpected network error while writing to 192.168.206.58: rc = -22 [ 1291.326986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1291.366032] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1291.435645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1291.460012] LNet: 68502:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1291.466839] LNetError: 68502:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1291.474663] LNet: Removed LNI 192.168.206.58@tcp [ 1291.483719] LNet: Removed LNI 192.168.206.58@tcp1 [ 1291.531596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1291.578987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1291.626684] LNet: Added LNI 192.168.206.58@tcp [8/256/0/180] [ 1292.454381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1292.482560] LNet: Added LNI 192.168.206.58@tcp1 [8/256/0/180] [ 1293.241596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.58@tcp [ 1293.268635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1293.319991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1293.362322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1293.393067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1293.425592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1293.473575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1293.527749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.206.58@tcp [ 1293.563484] LNetError: 68774:0:(api-ni.c:4478:lnet_ni_set_healthv()) manually adding local NI 192.168.206.58@tcp to recovery [ 1293.603433] LNet: There was an unexpected network error while writing to 192.168.206.58: rc = -22 [ 1295.992598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1299.748780] LNet: There was an unexpected network error while writing to 192.168.206.58: rc = -22 [ 1299.759098] LNet: Skipped 1 previous similar message [ 1304.056456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1304.089433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1304.138719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1311.636618] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 15:35:29 (1789673729) [ 1311.777466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1311.832836] LNet: 69476:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1311.851820] LNetError: 69476:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1311.869809] LNetError: 69476:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1311.879170] LNet: Removed LNI 192.168.206.58@tcp [ 1311.899924] LNet: Removed LNI 192.168.206.58@tcp1 [ 1311.955231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1312.018607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1312.051901] LNet: Added LNI 192.168.206.58@tcp [8/256/0/180] [ 1312.743467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1312.758966] LNet: Added LNI 192.168.206.58@tcp1 [8/256/0/180] [ 1313.651658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.58@tcp [ 1313.707488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1313.740334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1313.794704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1313.855768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1313.888967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1313.911596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.206.58@tcp --health 0 [ 1313.987749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1314.019280] LNet: 1 peer NIs in recovery (showing 1): 192.168.206.58@tcp [ 1314.024348] LNet: There was an unexpected network error while writing to 192.168.206.58: rc = -22 [ 1314.028462] LNet: Skipped 1 previous similar message [ 1329.436520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1329.578871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1329.645586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1329.696874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.206.58@tcp --health 500 [ 1329.879303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1336.549213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1336.624898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1336.650406] LNet: 69973:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1336.664504] LNetError: 69973:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1336.669871] LNetError: 69973:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1337.696140] LNet: Removed LNI 192.168.206.58@tcp [ 1337.709316] LNet: Removed LNI 192.168.206.58@tcp1 [ 1337.788525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1337.913250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1337.969573] LNet: Added LNI 192.168.206.58@tcp [8/256/0/180] [ 1338.890924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1338.915947] LNet: Added LNI 192.168.206.58@tcp1 [8/256/0/180] [ 1339.956821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.58@tcp [ 1340.002778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1340.083449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1340.133341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1340.166534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1340.197167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1340.228952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1340.264776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.206.58@tcp --health 0 [ 1341.922245] LNet: There was an unexpected network error while writing to 192.168.206.58: rc = -22 [ 1341.926773] LNet: Skipped 3 previous similar messages [ 1342.661762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1351.676224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1351.705781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1351.732523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1356.862790] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 15:36:15 (1789673775) [ 1357.374964] LNet: 71020:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1357.381738] LNetError: 71020:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1357.385981] LNetError: 71020:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1357.392865] LNet: Removed LNI 192.168.206.58@tcp [ 1357.417273] LNet: Removed LNI 192.168.206.58@tcp1 [ 1358.044774] Key type .llcrypt unregistered [ 1358.049174] Key type ._llcrypt unregistered [ 1362.888456] Key type ._llcrypt registered [ 1362.890723] Key type .llcrypt registered [ 1363.065563] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1363.079204] alg: No test for adler32 (adler32-zlib) [ 1363.929816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1363.971064] LNet: Added LNI 192.168.206.58@tcp [8/256/0/180] [ 1371.954556] Lustre: DEBUG MARKER: Initial discovery [ 1372.169954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.206.158@tcp [ 1375.368201] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1375.502342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1375.594894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.206.158@tcp [ 1375.644121] LNet: There was an unexpected network error while writing to 192.168.206.158: rc = -22 [ 1375.697175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1375.815663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.206.158@tcp -r 1 -m GET -l 3 [ 1375.880019] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.206.158@tcp -r 1 -m GET -e local_error [ 1375.976745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.206.158@tcp -r 1 -m PUT -l 6 [ 1376.298989] LNet: 1 local NIs in recovery (showing 1): 192.168.206.58@tcp [ 1378.672907] Lustre: DEBUG MARKER: Force oleg658-client.virtnet to discover 192.168.206.158@tcp (in background) [ 1378.766434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.206.158@tcp [ 1381.112790] Lustre: DEBUG MARKER: Force oleg658-server to discover 192.168.206.58@tcp [ 1381.858592] LNet: There was an unexpected network error while writing to 192.168.206.158: rc = -22 [ 1384.539360] Lustre: DEBUG MARKER: Wait for 71770 [ 1385.568409] LNet: There was an unexpected network error while writing to 192.168.206.158: rc = -22 [ 1386.082699] Lustre: DEBUG MARKER: Finished wait on 71770 [ 1386.455787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay del -a [ 1386.525883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1386.822282] LNet: 72228:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1386.828762] LNetError: 72228:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1386.851536] LNet: Removed LNI 192.168.206.58@tcp [ 1387.248411] Key type .llcrypt unregistered [ 1387.251068] Key type ._llcrypt unregistered [ 1404.986204] Key type ._llcrypt registered [ 1404.988485] Key type .llcrypt registered [ 1406.360514] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 15:37:04 (1789673824) [ 1406.781839] Key type .llcrypt unregistered [ 1406.783899] Key type ._llcrypt unregistered [ 1412.922348] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1413.137042] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1413.146610] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1413.151621] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1415.781995] Key type ._llcrypt registered [ 1415.785517] Key type .llcrypt registered [ 1416.040661] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1416.065269] alg: No test for adler32 (adler32-zlib) [ 1416.956176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1417.006911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1417.028790] LNet: Added LNI 192.168.206.58@tcp [8/256/0/180] [ 1417.617060] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1417.634544] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1425.618898] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 15:37:23 (1789673843) [ 1425.937191] LNet: 74294:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1425.946474] LNetError: 74294:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1425.950755] LNet: Removed LNI 10.1.2.3@tcp [ 1425.959641] LNet: Removed LNI 192.168.206.58@tcp [ 1426.987460] Key type .llcrypt unregistered [ 1426.992311] Key type ._llcrypt unregistered [ 1435.574823] Key type ._llcrypt registered [ 1435.579762] Key type .llcrypt registered [ 1435.819358] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1435.829542] alg: No test for adler32 (adler32-zlib) [ 1436.933591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1437.039625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1437.066355] LNet: Added LNI 192.168.206.58@tcp [8/256/0/180] [ 1438.001274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1438.020692] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1439.416650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.206.58@tcp [ 1440.485181] LNetError: Unexpected error -22 connecting to 192.168.206.58@tcp at host 192.168.206.58:988 [ 1446.077405] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 15:37:44 (1789673864) [ 1446.486401] LNet: 75424:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1446.498752] LNetError: 75424:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1446.508384] LNet: Removed LNI 10.1.2.3@tcp [ 1446.517737] LNet: Removed LNI 192.168.206.58@tcp [ 1447.332462] Key type .llcrypt unregistered [ 1447.337706] Key type ._llcrypt unregistered [ 1453.247687] Key type ._llcrypt registered [ 1453.249679] Key type .llcrypt registered [ 1453.365685] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1453.380755] alg: No test for adler32 (adler32-zlib) [ 1454.279605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1454.341816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1454.372607] LNet: Added LNI 192.168.206.58@tcp1 [8/256/0/180] [ 1455.187107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1455.211707] LNet: Added LNI 192.168.206.58@tcp2 [8/256/0/180] [ 1456.105025] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.206.58@tcp1 --nid 192.168.206.58@tcp2 [ 1460.479075] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.58@tcp1 192.168.206.58@tcp2 [ 1460.509292] LNetError: 76256:0:(lib-move.c:2094:lnet_handle_find_routed_path()) Unable to send message from 192.168.206.58@tcp1 to 192.168.206.58@tcp2 - Route table may be misconfigured [ 1460.535596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.58@tcp2 192.168.206.58@tcp1 [ 1465.686522] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 15:38:04 (1789673884) [ 1465.964362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1465.985840] LNet: 76830:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1465.990464] LNetError: 76830:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1465.994738] LNet: Removed LNI 192.168.206.58@tcp1 [ 1467.059738] LNet: Removed LNI 192.168.206.58@tcp2 [ 1467.101250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1467.175557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1467.217472] LNet: Added LNI 192.168.206.58@tcp [8/256/0/180] [ 1467.923462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1467.941228] LNet: Added LNI 192.168.206.58@tcp1 [8/256/0/180] [ 1468.684729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.58@tcp [ 1468.735985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.58@tcp 192.168.206.58@tcp [ 1468.770447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.58@tcp1 192.168.206.58@tcp1 [ 1468.813646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.206.58@tcp -d 192.168.206.58@tcp -e network_timeout [ 1468.862561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.206.58@tcp -d 192.168.206.58@tcp1 -e network_timeout [ 1468.904361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.206.58@tcp1 -d 192.168.206.58@tcp -e network_timeout [ 1468.928344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.206.58@tcp1 -d 192.168.206.58@tcp1 -e network_timeout [ 1468.952636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.58@tcp [ 1468.989174] LNet: There was an unexpected network error while writing to 192.168.206.58: rc = -22 [ 1469.016789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1469.125807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1469.207825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1469.219107] LNet: 1 local NIs in recovery (showing 1): 192.168.206.58@tcp [ 1474.588760] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 15:38:12 (1789673892) [ 1474.675312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1474.734844] LNet: 77687:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1474.741683] LNetError: 77687:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1474.749690] LNetError: 77687:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1474.756608] LNet: Removed LNI 192.168.206.58@tcp [ 1475.831715] LNet: Removed LNI 192.168.206.58@tcp1 [ 1475.893352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1475.998921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1476.202353] LNet: 77729:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1476.551662] Key type .llcrypt unregistered [ 1476.555084] Key type ._llcrypt unregistered [ 1486.856804] Key type ._llcrypt registered [ 1486.859899] Key type .llcrypt registered [ 1488.103272] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 15:38:26 (1789673906) [ 1488.479810] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1488.493412] alg: No test for adler32 (adler32-zlib) [ 1489.451461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1490.577694] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1492.022755] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 15:38:30 (1789673910) [ 1492.242947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1492.260307] LNet: 78823:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1492.288806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1492.342095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1492.362691] LNet: Added LNI 192.168.206.58@tcp [8/256/0/180] [ 1493.081029] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1493.112727] LNet: Added LNI 192.168.206.58@tcp1 [8/256/0/180] [ 1493.902839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.58@tcp [ 1493.971693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.58@tcp1 [ 1494.024964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.58@tcp1 [ 1499.789827] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 15:38:38 (1789673918) [ 1501.452559] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg658-server [ 1502.960443] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 15:38:41 (1789673921) [ 1504.542975] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg658-server [ 1506.173191] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 15:38:44 (1789673924) [ 1507.921527] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg658-server [ 1509.368564] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 15:38:47 (1789673927) [ 1511.325347] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg658-server [ 1512.972617] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 15:38:51 (1789673931) [ 1514.540850] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg658-server [ 1516.138936] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 15:38:54 (1789673934) [ 1517.684626] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg658-server [ 1519.382748] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 15:38:57 (1789673937) [ 1521.347635] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg658-server [ 1523.247895] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 15:39:01 (1789673941) [ 1524.996835] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg658-server [ 1526.554976] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 15:39:04 (1789673944) [ 1528.302537] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg658-server [ 1530.119593] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 15:39:08 (1789673948) [ 1530.223270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1530.269736] LNet: 82695:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1530.283666] LNetError: 82695:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1530.288962] LNet: Removed LNI 192.168.206.58@tcp [ 1530.325389] LNet: Removed LNI 192.168.206.58@tcp1 [ 1530.375520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1530.427261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1530.448947] LNet: Added LNI 192.168.206.58@tcp [8/256/0/180] [ 1531.350413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1531.466971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.58@tcp [ 1531.648630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1531.677725] LNet: 82847:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1531.683912] LNetError: 82847:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1531.692044] LNetError: 82847:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1532.714363] LNet: Removed LNI 192.168.206.58@tcp [ 1532.759932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1532.848822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1532.888668] LNet: Added LNI 192.168.206.58@tcp [8/256/0/180] [ 1533.657385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.58@tcp [ 1533.865403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1533.892179] LNet: 82992:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1533.901961] LNetError: 82992:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1533.926399] LNet: Removed LNI 192.168.206.58@tcp [ 1533.962905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1534.061526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1534.100612] LNet: Added LNI 192.168.206.58@tcp [8/256/0/180] [ 1534.809332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1534.891589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.58@tcp [ 1535.114448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1536.165765] LNet: Removed LNI 192.168.206.58@tcp [ 1536.252628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1536.344954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1536.373419] LNet: Added LNI 192.168.206.58@tcp [8/256/0/180] [ 1537.173324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.58@tcp [ 1537.407486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1537.431522] LNetError: 83285:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1537.441792] LNetError: 83285:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1537.465807] LNet: Removed LNI 192.168.206.58@tcp [ 1537.502179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1537.594067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1537.619524] LNet: Added LNI 192.168.206.58@tcp [8/256/0/180] [ 1538.434535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1538.532118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.58@tcp [ 1538.780868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1538.806259] LNet: 83432:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1538.810325] LNet: 83432:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 1539.878598] LNet: Removed LNI 192.168.206.58@tcp [ 1539.921313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1539.982653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1540.006049] LNet: Added LNI 192.168.206.58@tcp [8/256/0/180] [ 1540.635878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.58@tcp [ 1540.914531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1541.991571] LNet: Removed LNI 192.168.206.58@tcp [ 1542.017479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1542.092981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1542.127289] LNet: Added LNI 192.168.206.58@tcp [8/256/0/180] [ 1543.039362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1543.124845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.58@tcp [ 1543.397695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1543.419642] LNetError: 83725:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1543.426974] LNetError: 83725:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1543.443647] LNet: Removed LNI 192.168.206.58@tcp [ 1543.481641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1543.562322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1543.588947] LNet: Added LNI 192.168.206.58@tcp [8/256/0/180] [ 1544.502590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.58@tcp [ 1544.791156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1545.833363] LNet: Removed LNI 192.168.206.58@tcp [ 1545.868941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1545.946790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1545.964282] LNet: Added LNI 192.168.206.58@tcp [8/256/0/180] [ 1546.582413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1546.641391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.58@tcp [ 1546.881493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1546.925973] LNet: 84017:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1546.931834] LNet: 84017:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 1548.006480] LNet: Removed LNI 192.168.206.58@tcp [ 1548.069382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1548.113877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1548.148127] LNet: Added LNI 192.168.206.58@tcp [8/256/0/180] [ 1548.885828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.58@tcp [ 1549.221903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1550.316256] LNet: Removed LNI 192.168.206.58@tcp [ 1550.363647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1550.438966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1550.467697] LNet: Added LNI 192.168.206.58@tcp [8/256/0/180] [ 1551.193137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1551.256687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.58@tcp [ 1551.521248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1551.551842] LNetError: 84310:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1551.566598] LNetError: 84310:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 1551.585250] LNet: Removed LNI 192.168.206.58@tcp [ 1551.604394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1551.662806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1551.692509] LNet: Added LNI 192.168.206.58@tcp [8/256/0/180] [ 1552.508835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.58@tcp [ 1552.807838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1553.896227] LNet: Removed LNI 192.168.206.58@tcp [ 1553.931561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1554.029735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1554.058624] LNet: Added LNI 192.168.206.58@tcp [8/256/0/180] [ 1554.939736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1555.003768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1555.036578] LNet: Removed LNI 192.168.206.58@tcp [ 1555.057788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1555.146882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1555.192235] LNet: Added LNI 192.168.206.58@tcp [8/256/0/180] [ 1556.279694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1556.422972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1556.469608] LNet: Removed LNI 192.168.206.58@tcp [ 1556.498763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1556.577862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 128 [ 1556.698556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1556.769047] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1556.828581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 1000000000000000000000000 [ 1556.926399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1556.978332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1557.076132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer foo [ 1557.236194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1557.343827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1557.402871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 8 [ 1557.420410] LNet: Added LNI 192.168.206.58@tcp [8/256/0/180] [ 1558.224869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1558.274333] LNet: Removed LNI 192.168.206.58@tcp [ 1558.290451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1558.343795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1564.931925] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 15:39:43 (1789673983) [ 1565.103298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1565.172263] LNet: 85475:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1565.179850] LNet: 85475:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 9 previous similar messages [ 1565.205327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1565.253397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1565.281420] LNet: Added LNI 192.168.206.58@tcp231 [8/256/0/180] [ 1566.391513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1566.451854] LNet: Removed LNI 192.168.206.58@tcp231 [ 1566.474532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1566.569520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1566.589179] LNet: Added LNI 192.168.206.58@tcp231 [8/256/0/180] [ 1566.593035] LNet: 85622:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1567.088779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1567.117344] LNet: Removed LNI 192.168.206.58@tcp231 [ 1567.132387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1567.161512] LNet: Added LNI 192.168.206.58@tcp231 [8/256/0/180] [ 1568.032373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1568.047953] LNetError: 85766:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1568.053445] LNetError: 85766:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 1568.069638] LNet: Removed LNI 192.168.206.58@tcp231 [ 1568.096366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1568.177249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1568.202367] LNet: Added LNI 192.168.206.58@tcp231 [8/256/0/0] [ 1568.206604] LNet: 85774:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1568.616543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1568.652898] LNet: Removed LNI 192.168.206.58@tcp231 [ 1568.683720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1568.716715] LNet: Added LNI 192.168.206.58@tcp231 [8/256/0/0] [ 1570.043102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1570.074979] LNet: Removed LNI 192.168.206.58@tcp231 [ 1570.108609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1570.203719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1570.251143] LNet: Added LNI 192.168.206.58@tcp231 [8/256/0/60] [ 1570.260751] LNet: 85925:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1570.818772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1570.844127] LNet: Removed LNI 192.168.206.58@tcp231 [ 1570.868458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1570.895941] LNet: Added LNI 192.168.206.58@tcp231 [8/256/0/60] [ 1571.986034] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1572.011417] LNet: Removed LNI 192.168.206.58@tcp231 [ 1572.031681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1572.115302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1572.149995] LNet: Added LNI 192.168.206.58@tcp231 [8/256/0/180] [ 1572.563422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1572.598612] LNet: Removed LNI 192.168.206.58@tcp231 [ 1572.610878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1572.636425] LNet: Added LNI 192.168.206.58@tcp231 [8/256/0/180] [ 1579.720137] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 15:39:57 (1789673997) [ 1579.817759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1579.856192] LNet: Removed LNI 192.168.206.58@tcp231 [ 1579.879570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1579.946417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1579.973659] LNet: Added LNI 192.168.206.58@tcp232 [8/256/0/180] [ 1580.600759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1580.652658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1580.696957] LNet: Removed LNI 192.168.206.58@tcp232 [ 1580.713423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1580.740283] LNet: Added LNI 192.168.206.58@tcp232 [8/256/0/180] [ 1581.407402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1581.445509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1581.473741] LNet: Removed LNI 192.168.206.58@tcp232 [ 1581.491879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1581.525218] LNet: Added LNI 192.168.206.58@tcp232 [8/256/0/180] [ 1582.261345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1582.299761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1582.338255] LNet: Removed LNI 192.168.206.58@tcp232 [ 1582.357636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1582.393089] LNet: Added LNI 192.168.206.58@tcp232 [8/256/0/180] [ 1583.139118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1583.168609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1583.196450] LNet: Removed LNI 192.168.206.58@tcp232 [ 1583.213755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1589.721761] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 15:40:07 (1789674007) [ 1590.810068] Key type .llcrypt unregistered [ 1590.814848] Key type ._llcrypt unregistered [ 1597.104189] Key type ._llcrypt registered [ 1597.106881] Key type .llcrypt registered [ 1597.336714] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1597.362167] alg: No test for adler32 (adler32-zlib) [ 1598.332470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1598.360473] LNet: Added LNI 192.168.206.58@tcp [8/256/0/180] [ 1606.156284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.158@tcp [ 1615.631559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1615.660136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1615.677341] LNet: Added LNI 192.168.206.58@tcp1 [8/256/0/180] [ 1617.096271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.58@tcp -d 192.168.206.158@tcp -m GET -e local_interrupt -r 2 [ 1617.124273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.58@tcp -d 192.168.206.158@tcp1 -m GET -e local_interrupt -r 2 [ 1617.161483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.58@tcp -d 192.168.206.58@tcp -m GET -e local_interrupt -r 2 [ 1617.201518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.58@tcp -d 192.168.206.58@tcp1 -m GET -e local_interrupt -r 2 [ 1617.224665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.58@tcp1 -d 192.168.206.158@tcp -m GET -e local_interrupt -r 2 [ 1617.249946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.58@tcp1 -d 192.168.206.158@tcp1 -m GET -e local_interrupt -r 2 [ 1617.274977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.58@tcp1 -d 192.168.206.58@tcp -m GET -e local_interrupt -r 2 [ 1617.302436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.58@tcp1 -d 192.168.206.58@tcp1 -m GET -e local_interrupt -r 2 [ 1617.343826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.58@tcp 192.168.206.158@tcp [ 1617.373992] LNet: There was an unexpected network error while writing to 192.168.206.158: rc = -22 [ 1617.380211] LNet: 1 local NIs in recovery (showing 1): 192.168.206.58@tcp [ 1617.605896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1617.628615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1617.659941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1617.698334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1617.736074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1617.762472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1617.801803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1617.840391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1617.882353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1617.906476] LNet: 88575:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1617.927433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1617.970512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1618.029281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1618.086511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1618.156320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1618.198280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1618.777551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.58@tcp -d 192.168.206.158@tcp -m GET -e local_dropped -r 2 [ 1618.823900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.58@tcp -d 192.168.206.158@tcp1 -m GET -e local_dropped -r 2 [ 1618.883132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.58@tcp -d 192.168.206.58@tcp -m GET -e local_dropped -r 2 [ 1618.935707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.58@tcp -d 192.168.206.58@tcp1 -m GET -e local_dropped -r 2 [ 1618.979977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.58@tcp1 -d 192.168.206.158@tcp -m GET -e local_dropped -r 2 [ 1619.023601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.58@tcp1 -d 192.168.206.158@tcp1 -m GET -e local_dropped -r 2 [ 1619.080794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.58@tcp1 -d 192.168.206.58@tcp -m GET -e local_dropped -r 2 [ 1619.140543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.58@tcp1 -d 192.168.206.58@tcp1 -m GET -e local_dropped -r 2 [ 1619.188643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.58@tcp 192.168.206.158@tcp [ 1619.227775] LNet: There was an unexpected network error while writing to 192.168.206.158: rc = -22 [ 1619.485945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1619.529947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1619.582438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1619.636344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1619.687571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1619.732851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1619.787068] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1619.835793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1619.872258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1619.892060] LNet: 88698:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1619.909556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1619.932687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1619.959459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1619.986192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1620.024511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1620.046521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1620.507969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.58@tcp -d 192.168.206.158@tcp -m GET -e local_aborted -r 2 [ 1620.545314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.58@tcp -d 192.168.206.158@tcp1 -m GET -e local_aborted -r 2 [ 1620.584560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.58@tcp -d 192.168.206.58@tcp -m GET -e local_aborted -r 2 [ 1620.618125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.58@tcp -d 192.168.206.58@tcp1 -m GET -e local_aborted -r 2 [ 1620.647596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.58@tcp1 -d 192.168.206.158@tcp -m GET -e local_aborted -r 2 [ 1620.678578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.58@tcp1 -d 192.168.206.158@tcp1 -m GET -e local_aborted -r 2 [ 1620.703480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.58@tcp1 -d 192.168.206.58@tcp -m GET -e local_aborted -r 2 [ 1620.741938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.58@tcp1 -d 192.168.206.58@tcp1 -m GET -e local_aborted -r 2 [ 1620.774096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.58@tcp 192.168.206.158@tcp [ 1620.792347] LNet: There was an unexpected network error while writing to 192.168.206.158: rc = -22 [ 1621.011181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1621.055376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1621.092857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1621.141500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1621.195462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1621.238934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1621.298430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1621.343908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1621.375854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1621.408237] LNet: 88820:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1621.446926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1621.485521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1621.531257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1621.576970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1621.615296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1621.650232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1622.239552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.58@tcp -d 192.168.206.158@tcp -m GET -e local_no_route -r 2 [ 1622.310535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.58@tcp -d 192.168.206.158@tcp1 -m GET -e local_no_route -r 2 [ 1622.372455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.58@tcp -d 192.168.206.58@tcp -m GET -e local_no_route -r 2 [ 1622.419948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.58@tcp -d 192.168.206.58@tcp1 -m GET -e local_no_route -r 2 [ 1622.468108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.58@tcp1 -d 192.168.206.158@tcp -m GET -e local_no_route -r 2 [ 1622.530693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.58@tcp1 -d 192.168.206.158@tcp1 -m GET -e local_no_route -r 2 [ 1622.576264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.58@tcp1 -d 192.168.206.58@tcp -m GET -e local_no_route -r 2 [ 1622.645716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.58@tcp1 -d 192.168.206.58@tcp1 -m GET -e local_no_route -r 2 [ 1622.706639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.58@tcp 192.168.206.158@tcp [ 1623.043312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1623.090580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1623.120816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1623.166329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1623.223440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1623.267648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1623.304982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1623.347550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1623.382501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1623.412465] LNet: 88942:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1623.434718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1623.474694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1623.529640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1623.568307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1623.614379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1623.665979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1624.260318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.58@tcp -d 192.168.206.158@tcp -m GET -e local_timeout -r 2 [ 1624.312914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.58@tcp -d 192.168.206.158@tcp1 -m GET -e local_timeout -r 2 [ 1624.371195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.58@tcp -d 192.168.206.58@tcp -m GET -e local_timeout -r 2 [ 1624.410889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.58@tcp -d 192.168.206.58@tcp1 -m GET -e local_timeout -r 2 [ 1624.455848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.58@tcp1 -d 192.168.206.158@tcp -m GET -e local_timeout -r 2 [ 1624.503963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.58@tcp1 -d 192.168.206.158@tcp1 -m GET -e local_timeout -r 2 [ 1624.554592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.58@tcp1 -d 192.168.206.58@tcp -m GET -e local_timeout -r 2 [ 1624.595500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.58@tcp1 -d 192.168.206.58@tcp1 -m GET -e local_timeout -r 2 [ 1624.632222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.58@tcp 192.168.206.158@tcp [ 1624.656442] LNet: There was an unexpected network error while writing to 192.168.206.158: rc = -22 [ 1624.667220] LNet: Skipped 1 previous similar message [ 1624.975676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1625.021331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1625.074938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1625.126247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1625.170492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1625.231923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1625.277428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1625.353435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1625.432324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1625.474163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1625.524588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1625.553923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1625.588717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1625.640521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1625.701266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1626.257892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.58@tcp -d 192.168.206.158@tcp -m GET -e remote_dropped -r 2 [ 1626.296878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.58@tcp -d 192.168.206.158@tcp1 -m GET -e remote_dropped -r 2 [ 1626.332358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.58@tcp -d 192.168.206.58@tcp -m GET -e remote_dropped -r 2 [ 1626.361141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.58@tcp -d 192.168.206.58@tcp1 -m GET -e remote_dropped -r 2 [ 1626.388086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.58@tcp1 -d 192.168.206.158@tcp -m GET -e remote_dropped -r 2 [ 1626.426205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.58@tcp1 -d 192.168.206.158@tcp1 -m GET -e remote_dropped -r 2 [ 1626.477295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.58@tcp1 -d 192.168.206.58@tcp -m GET -e remote_dropped -r 2 [ 1626.534496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.58@tcp1 -d 192.168.206.58@tcp1 -m GET -e remote_dropped -r 2 [ 1626.569588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.58@tcp 192.168.206.158@tcp [ 1626.601188] LNet: 1 peer NIs in recovery (showing 1): 192.168.206.158@tcp [ 1626.846881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1626.891120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1626.944235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1626.983335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1627.049190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1627.093941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1627.137491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1627.175769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1627.214881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1627.253344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1627.312182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1627.346518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1627.407430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1627.466544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1627.511297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1633.153202] LNet: 89261:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1633.159639] LNetError: 89261:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1633.166339] LNet: Removed LNI 192.168.206.58@tcp [ 1633.181819] LNet: Removed LNI 192.168.206.58@tcp1 [ 1633.581529] Key type .llcrypt unregistered [ 1633.586395] Key type ._llcrypt unregistered [ 1651.566859] Key type ._llcrypt registered [ 1651.570974] Key type .llcrypt registered [ 1652.882720] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 15:41:10 (1789674070) [ 1653.313720] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1653.331731] alg: No test for adler32 (adler32-zlib) [ 1654.307925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1654.381318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1654.422055] LNet: Added LNI 192.168.206.58@tcp [8/256/0/180] [ 1655.277449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1655.305326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.206.58@tcp [ 1655.320305] LNetError: 90237:0:(api-ni.c:4478:lnet_ni_set_healthv()) manually adding local NI 192.168.206.58@tcp to recovery [ 1655.393263] LNet: 1 local NIs in recovery (showing 1): 192.168.206.58@tcp [ 1655.423441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.206.58@tcp [ 1655.533595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1661.045151] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 15:41:19 (1789674079) [ 1661.167362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1661.194379] LNet: 90825:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1661.210399] LNetError: 90825:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1661.237501] LNet: Removed LNI 192.168.206.58@tcp [ 1661.260143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1661.308715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1661.330813] LNet: Added LNI 192.168.206.58@tcp [8/256/0/180] [ 1662.012445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1662.030604] LNet: Added LNI 192.168.206.58@tcp1 [8/256/0/180] [ 1662.854587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.206.58@tcp [ 1662.877495] LNetError: 91092:0:(api-ni.c:4478:lnet_ni_set_healthv()) manually adding local NI 192.168.206.58@tcp to recovery [ 1663.329245] LNet: There was an unexpected network error while writing to 192.168.206.58: rc = -22 [ 1664.089987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.206.58@tcp [ 1672.751710] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 15:41:30 (1789674090) [ 1674.580838] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg658-server [ 1676.349144] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 15:41:34 (1789674094) [ 1678.044877] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg658-server [ 1679.898899] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 15:41:37 (1789674097) [ 1681.607455] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg658-server [ 1683.279152] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 15:41:41 (1789674101) [ 1683.437452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1683.475297] LNet: 92720:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1683.489206] LNetError: 92720:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1683.497656] LNet: Removed LNI 192.168.206.58@tcp [ 1683.513485] LNet: Removed LNI 192.168.206.58@tcp1 [ 1683.568857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1683.654739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1683.674053] LNet: Added LNI 192.168.206.58@tcp [8/256/0/180] [ 1684.515884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1684.585663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1684.648821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 1684.825780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1684.850475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 1684.963737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1690.276414] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 15:41:48 (1789674108) [ 1690.361585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1690.387665] LNet: 93444:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1690.409248] LNetError: 93444:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1690.425817] LNetError: 93444:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1690.450798] LNet: Removed LNI 192.168.206.58@tcp [ 1690.482682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1690.541610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1690.570277] LNet: Added LNI 192.168.206.58@tcp [8/256/0/180] [ 1697.437383] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 15:41:55 (1789674115) [ 1697.579870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1697.615315] LNet: 94152:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1697.640019] LNetError: 94152:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1697.669865] LNet: Removed LNI 192.168.206.58@tcp [ 1697.733988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1697.797734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1697.832996] LNet: Added LNI 192.168.206.58@tcp [8/256/0/180] [ 1706.381303] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 15:42:04 (1789674124) [ 1707.816937] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 1709.357382] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 15:42:07 (1789674127) [ 1710.009747] LNet: 95191:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1710.014108] LNetError: 95191:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1710.028943] LNet: Removed LNI 192.168.206.58@tcp [ 1710.533914] Key type .llcrypt unregistered [ 1710.538216] Key type ._llcrypt unregistered [ 1716.563941] Key type ._llcrypt registered [ 1716.565469] Key type .llcrypt registered [ 1716.845816] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1716.867700] alg: No test for adler32 (adler32-zlib) [ 1717.804560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1717.829520] LNet: Added LNI 192.168.206.58@tcp [8/256/0/180] [ 1725.801189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.158@tcp [ 1731.627464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.206.158@tcp [ 1732.128150] LNet: 95605:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1732.147387] LNetError: 95605:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1732.171493] LNet: Removed LNI 192.168.206.58@tcp [ 1732.654853] Key type .llcrypt unregistered [ 1732.662887] Key type ._llcrypt unregistered [ 1744.422389] Key type ._llcrypt registered [ 1744.424646] Key type .llcrypt registered [ 1745.652864] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 1747.256769] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 1749.001120] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 15:42:47 (1789674167) [ 1750.676555] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 [ 1752.163682] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 15:42:50 (1789674170) [ 1754.018464] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg658-server [ 1755.707815] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 15:42:53 (1789674173) [ 1757.458751] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg658-server [ 1759.318784] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 15:42:57 (1789674177) [ 1761.066493] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1761.083209] alg: No test for adler32 (adler32-zlib) [ 1762.197304] Lustre: Lustre: Build Version: 2.17.58_39_g9373f0b [ 1762.510156] LNet: Added LNI 192.168.206.58@tcp [8/256/0/180] [ 1764.224210] Key type lgssc registered [ 1765.533868] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1786.647695] Key type lgssc unregistered [ 1786.909657] LNet: 98778:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1786.915864] LNetError: 98778:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1786.941224] LNet: Removed LNI 192.168.206.58@tcp [ 1787.728655] Key type .llcrypt unregistered [ 1787.730356] Key type ._llcrypt unregistered [ 1803.816918] Key type ._llcrypt registered [ 1803.819057] Key type .llcrypt registered [ 1805.237453] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 15:43:43 (1789674223) [ 1805.538432] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1805.555591] alg: No test for adler32 (adler32-zlib) [ 1806.470531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1806.521068] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1806.557709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1806.588679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1806.623666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1812.229698] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 15:43:50 (1789674230) [ 1812.438620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1812.468151] LNet: 100191:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1812.490540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1812.540920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1812.559252] LNet: Added LNI 192.168.206.58@tcp [8/256/0/180] [ 1813.403616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1813.441903] LNet: 100335:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1813.452500] LNetError: 100335:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1813.477836] LNet: Removed LNI 192.168.206.58@tcp [ 1813.501556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1813.535912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1813.556242] LNet: Added LNI 192.168.206.58@tcp [8/256/0/180] [ 1814.050938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1814.070360] LNetError: 100363:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1814.082782] LNet: Removed LNI 192.168.206.58@tcp [ 1814.099097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1814.144726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1814.168464] LNet: Added LNI 192.168.206.58@tcp [8/256/0/180] [ 1819.963979] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 15:43:58 (1789674238) [ 1820.285731] LNet: 100953:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1820.295529] LNet: 100953:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1820.309847] LNetError: 100953:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1820.336532] LNet: Removed LNI 192.168.206.58@tcp [ 1821.016966] Key type .llcrypt unregistered [ 1821.019205] Key type ._llcrypt unregistered [ 1827.242819] Key type ._llcrypt registered [ 1827.245243] Key type .llcrypt registered [ 1827.441515] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1827.470854] alg: No test for adler32 (adler32-zlib) [ 1828.382650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1828.452211] LNet: Added LNI 192.168.206.58@tcp [8/256/0/180] [ 1828.463549] LNet: *** cfs_fail_loc=e003, val=100*** [ 1829.208687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.58@tcp [ 1829.239813] LNet: *** cfs_fail_loc=e003, val=100*** [ 1829.243729] LNet: Skipped 2 previous similar messages [ 1829.262297] Ping reply received with mapping: [ 1829.264069] found metadata reply NI[192.168.206.58@tcp] [ 1829.271056] 3a ce a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 1829.280929] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 1829.284289] 5a 5a 5a 5a 5a 5a 5a 5a [ 1829.408479] LNet: 101347:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1829.427707] LNetError: 101347:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1829.463210] LNet: Removed LNI 192.168.206.58@tcp [ 1830.048513] Key type .llcrypt unregistered [ 1830.050225] Key type ._llcrypt unregistered [ 1840.527233] Key type ._llcrypt registered [ 1840.535756] Key type .llcrypt registered [ 1841.660733] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 15:44:19 (1789674259) [ 1842.154541] Key type .llcrypt unregistered [ 1842.158359] Key type ._llcrypt unregistered [ 1848.307985] Key type ._llcrypt registered [ 1848.309716] Key type .llcrypt registered [ 1848.492305] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1848.505521] alg: No test for adler32 (adler32-zlib) [ 1849.590187] LNetError: 102398:0:(api-ni.c:2855:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 1849.977553] Key type .llcrypt unregistered [ 1849.980407] Key type ._llcrypt unregistered [ 1855.149813] Key type ._llcrypt registered [ 1855.154364] Key type .llcrypt registered [ 1856.688716] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 15:44:34 (1789674274) [ 1857.231810] Key type .llcrypt unregistered [ 1857.233813] Key type ._llcrypt unregistered [ 1863.432877] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1863.445834] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1864.802676] Key type ._llcrypt registered [ 1864.805272] Key type .llcrypt registered [ 1865.009771] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1865.020835] alg: No test for adler32 (adler32-zlib) [ 1865.923905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1866.025390] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1874.738533] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 15:44:52 (1789674292) [ 1874.897512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1874.927650] LNet: 104159:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1874.945446] LNetError: 104159:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1874.978843] LNet: Removed LNI 10.1.2.3@tcp [ 1875.020493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1875.669478] Key type .llcrypt unregistered [ 1875.673497] Key type ._llcrypt unregistered [ 1882.168627] Key type ._llcrypt registered [ 1882.170507] Key type .llcrypt registered [ 1882.487627] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1882.553273] alg: No test for adler32 (adler32-zlib) [ 1883.586429] LNet: Added LNI 192.168.206.58@tcp [8/256/0/180] [ 1883.688718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.58@tcp [ 1883.896303] LNet: 104448:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1883.917172] LNetError: 104448:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1883.942507] LNet: Removed LNI 192.168.206.58@tcp [ 1884.714438] Key type .llcrypt unregistered [ 1884.718587] Key type ._llcrypt unregistered [ 1894.969240] Key type ._llcrypt registered [ 1894.971939] Key type .llcrypt registered [ 1896.129655] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 15:45:14 (1789674314) [ 1896.866340] Key type .llcrypt unregistered [ 1896.869617] Key type ._llcrypt unregistered [ 1903.516801] Key type ._llcrypt registered [ 1903.518819] Key type .llcrypt registered [ 1903.745624] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1903.760472] alg: No test for adler32 (adler32-zlib) [ 1911.199344] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 15:45:29 (1789674329) [ 1911.341957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1911.377801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1911.446179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1911.464422] LNet: Added LNI 192.168.206.58@tcp [8/256/0/180] [ 1912.259697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1912.294742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 1912.325300] LNet: Removed LNI 192.168.206.58@tcp [ 1912.540970] LNet: 106187:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1912.897330] Key type .llcrypt unregistered [ 1912.899774] Key type ._llcrypt unregistered [ 1923.615530] Key type ._llcrypt registered [ 1923.617973] Key type .llcrypt registered [ 1924.777234] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 15:45:42 (1789674342) [ 1925.045948] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1925.061690] alg: No test for adler32 (adler32-zlib) [ 1925.980138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1926.081570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1926.117714] LNet: Added LNI 192.168.206.58@tcp [8/256/0/180] [ 1926.828550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.58@tcp [ 1926.884657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.206.58@tcp [ 1926.915440] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 1926.922450] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 1926.927688] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1926, 0, 0 [ 1926.937120] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 1926.944937] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 1926.952090] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 1926.958444] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1926, 0, 0 [ 1926.965626] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 1926.971192] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1926.977572] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 1926.985434] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1926, 0, 1976 [ 1926.995614] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 1927.001846] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1927.009866] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 1927.015793] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1926, 0, 1976 [ 1927.021607] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 1932.015637] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 15:45:50 (1789674350) [ 1932.333634] LNet: 107725:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1932.339491] LNetError: 107725:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1932.355816] LNet: Removed LNI 192.168.206.58@tcp [ 1932.903378] Key type .llcrypt unregistered [ 1932.907046] Key type ._llcrypt unregistered [ 1938.712166] Key type ._llcrypt registered [ 1938.716292] Key type .llcrypt registered [ 1938.972477] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1938.989928] alg: No test for adler32 (adler32-zlib) [ 1939.984906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1940.006393] LNet: Added LNI 192.168.206.58@tcp [8/256/0/180] [ 1946.599463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.158@tcp [ 1953.591477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1953.639395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1953.659453] LNet: Added LNI 192.168.206.58@tcp1 [8/256/0/180] [ 1954.369199] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1954.531932] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1954.541955] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1954.546802] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1956.804515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 1956.830607] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 1964.978143] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 15:46:23 (1789674383) [ 1965.168826] LNet: 109245:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1965.176925] LNetError: 109245:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1965.184425] LNet: Removed LNI 192.168.206.58@tcp [ 1966.240206] LNet: Removed LNI 192.168.206.58@tcp1 [ 1966.242486] LNetError: 109245:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1966.255130] LNetError: 109245:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1966.290922] LNet: Removed LNI 10.1.2.3@tcp99 [ 1966.973724] Key type .llcrypt unregistered [ 1966.984549] Key type ._llcrypt unregistered [ 1972.710465] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1972.714036] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1975.240578] Key type ._llcrypt registered [ 1975.242979] Key type .llcrypt registered [ 1975.403768] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1975.422374] alg: No test for adler32 (adler32-zlib) [ 1976.390457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1976.443643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1976.465881] LNet: Added LNI 192.168.206.58@tcp [8/256/0/180] [ 1977.085679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1977.113603] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1978.918251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 1978.971182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1979.021341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 1979.053404] LNetError: 109987:0:(peer.c:2366:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 1979.081235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1979.133190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.58@tcp [ 1979.608403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1979.713238] LNet: 110007:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1979.718795] LNetError: 110007:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1979.724498] LNet: Removed LNI 10.1.2.3@tcp [ 1980.777643] LNet: Removed LNI 192.168.206.58@tcp [ 1981.380385] Key type .llcrypt unregistered [ 1981.383911] Key type ._llcrypt unregistered [ 1992.378816] Key type ._llcrypt registered [ 1992.380577] Key type .llcrypt registered [ 1993.563849] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 15:46:51 (1789674411) [ 1993.982505] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1994.002145] alg: No test for adler32 (adler32-zlib) [ 1994.884274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1994.919325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1994.937332] LNet: Added LNI 192.168.206.58@tcp [8/256/0/180] [ 1995.659032] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.58@tcp [ 1995.751973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg658-client@tcp [ 2001.455892] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 15:46:59 (1789674419) [ 2001.537957] LNet: 111528:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2001.562307] LNetError: 111528:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2001.580741] LNet: Removed LNI 192.168.206.58@tcp [ 2002.204357] Key type .llcrypt unregistered [ 2002.209391] Key type ._llcrypt unregistered [ 2008.449380] Key type ._llcrypt registered [ 2008.452472] Key type .llcrypt registered [ 2008.587502] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2008.603440] alg: No test for adler32 (adler32-zlib) [ 2009.513890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 2009.543239] LNet: Added LNI 192.168.206.58@tcp [8/256/0/180] [ 2009.570242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 2015.798702] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 15:47:13 (1789674433) [ 2017.412280] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 2018.981759] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 15:47:17 (1789674437) [ 2020.516442] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 2022.098440] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 15:47:20 (1789674440) [ 2022.263188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2022.304387] LNet: 112952:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2022.316668] LNetError: 112952:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2022.337908] LNet: Removed LNI 192.168.206.58@tcp [ 2022.373928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2022.432888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2022.470268] LNet: Added LNI 192.168.206.58@tcp [8/256/0/180] [ 2023.247358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2023.292953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2029.211804] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 15:47:27 (1789674447) [ 2029.480979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2029.510332] LNet: 113665:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2029.520323] LNetError: 113665:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2029.542616] LNet: Removed LNI 192.168.206.58@tcp [ 2029.579988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2029.635246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 2029.666965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2030.200359] Key type .llcrypt unregistered [ 2030.202698] Key type ._llcrypt unregistered [ 2040.812792] Key type ._llcrypt registered [ 2040.815278] Key type .llcrypt registered [ 2042.111932] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 15:47:40 (1789674460) [ 2042.443568] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2042.458310] alg: No test for adler32 (adler32-zlib) [ 2043.383935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2043.441671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2043.464250] LNet: Added LNI 192.168.206.58@tcp [8/256/0/180] [ 2044.149586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2044.190622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.58@tcp [ 2050.469957] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 15:47:48 (1789674468) [ 2050.561878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2050.615233] LNet: 115211:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2050.626642] LNetError: 115211:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2050.658096] LNet: Removed LNI 192.168.206.58@tcp [ 2050.703206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2050.814645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 2050.901533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 2057.137411] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 15:47:55 (1789674475) [ 2057.329867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2057.349962] LNet: 115799:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2057.367731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2057.480926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2057.905411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2057.936450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2057.992846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 2058.304231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2058.334784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2058.417623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2058.449479] LNet: 115832:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2058.477963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2058.552880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2059.025416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2059.069465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2059.142146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 2059.604948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2059.640495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2066.126638] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 15:48:04 (1789674484) [ 2066.260527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2066.292209] LNet: 116428:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2066.317340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2066.379947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2066.422235] LNet: Added LNI 192.168.206.58@tcp [8/256/0/180] [ 2067.150352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2067.168835] LNet: Added LNI 192.168.206.58@tcp2 [8/256/0/180] [ 2067.864110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2067.909650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.58@tcp [ 2068.105253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2068.132346] LNetError: 116712:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2068.137126] LNet: Removed LNI 192.168.206.58@tcp [ 2068.159646] LNet: Removed LNI 192.168.206.58@tcp2 [ 2068.199125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2068.264271] LNet: Added LNI 192.168.206.58@tcp [8/256/0/180] [ 2068.274033] LNet: Added LNI 192.168.206.58@tcp2 [8/256/0/180] [ 2068.280939] LNet: 116718:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2068.777911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.58@tcp [ 2068.941767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2069.984167] LNet: Removed LNI 192.168.206.58@tcp [ 2069.988229] LNetError: 116755:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2069.996157] LNetError: 116755:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 2070.010822] LNet: Removed LNI 192.168.206.58@tcp2 [ 2070.035362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2070.116730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2070.157760] LNet: Added LNI 192.168.206.58@tcp [8/256/0/180] [ 2070.768324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2070.789143] LNet: Added LNI 192.168.206.58@tcp2 [8/256/0/180] [ 2071.420845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.58@tcp [ 2071.475310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2071.541349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.58@tcp [ 2071.750305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2071.768266] LNet: 117040:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2071.772681] LNet: 117040:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 2072.801612] LNet: Removed LNI 192.168.206.58@tcp [ 2072.804558] LNetError: 117040:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2072.808954] LNetError: 117040:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2072.831666] LNet: Removed LNI 192.168.206.58@tcp2 [ 2072.870019] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2072.958913] LNet: Added LNI 192.168.206.58@tcp [8/256/0/180] [ 2072.967453] LNet: Added LNI 192.168.206.58@tcp2 [8/256/0/180] [ 2072.973312] LNet: 117046:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2073.374693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.58@tcp [ 2073.547955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2073.566789] LNet: Removed LNI 192.168.206.58@tcp [ 2074.604603] LNet: Removed LNI 192.168.206.58@tcp2 [ 2074.640754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2074.694803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2074.713455] LNet: Added LNI 192.168.206.58@tcp [8/256/0/180] [ 2075.212467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2075.227918] LNet: Added LNI 192.168.206.58@tcp2 [8/256/0/180] [ 2075.941581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2075.992458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.58@tcp [ 2076.184688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2077.280156] LNet: Removed LNI 192.168.206.58@tcp [ 2077.282643] LNetError: 117363:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2077.293275] LNetError: 117363:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 2077.314257] LNet: Removed LNI 192.168.206.58@tcp2 [ 2077.348335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2077.447301] LNet: Added LNI 192.168.206.58@tcp [8/256/0/180] [ 2077.455106] LNet: Added LNI 192.168.206.58@tcp2 [8/256/0/180] [ 2077.463078] LNet: 117369:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2077.995175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.58@tcp [ 2078.190236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2079.265617] LNet: Removed LNI 192.168.206.58@tcp [ 2079.281057] LNet: Removed LNI 192.168.206.58@tcp2 [ 2079.308904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2079.364637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2079.393203] LNet: Added LNI 192.168.206.58@tcp [8/256/0/180] [ 2079.974610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2079.993231] LNet: Added LNI 192.168.206.58@tcp2 [8/256/0/180] [ 2080.711788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.58@tcp [ 2080.754992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2080.857091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.58@tcp [ 2081.082467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2081.110718] LNet: 117691:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2081.116436] LNet: 117691:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 2082.144173] LNet: Removed LNI 192.168.206.58@tcp [ 2083.179569] LNet: Removed LNI 192.168.206.58@tcp2 [ 2083.208812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2083.261717] LNet: Added LNI 192.168.206.58@tcp [8/256/0/180] [ 2083.266677] LNet: Added LNI 192.168.206.58@tcp2 [8/256/0/180] [ 2083.271683] LNet: 117697:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2083.769781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.58@tcp [ 2083.986355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2084.023445] LNet: Removed LNI 192.168.206.58@tcp [ 2085.100796] LNet: Removed LNI 192.168.206.58@tcp2 [ 2085.174333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2085.232575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2085.259891] LNet: Added LNI 192.168.206.58@tcp [8/256/0/180] [ 2085.901726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2085.923905] LNet: Added LNI 192.168.206.58@tcp2 [8/256/0/180] [ 2086.638745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2086.665299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2086.714114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.58@tcp [ 2087.041557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2087.060954] LNetError: 118031:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2087.065968] LNetError: 118031:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 6 previous similar messages [ 2088.099103] LNet: Removed LNI 192.168.206.58@tcp [ 2088.112600] LNet: Removed LNI 192.168.206.58@tcp2 [ 2088.146670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2088.214316] LNet: Added LNI 192.168.206.58@tcp [8/256/0/180] [ 2088.223035] LNet: Added LNI 192.168.206.58@tcp2 [8/256/0/180] [ 2088.230065] LNet: 118037:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2088.651414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.58@tcp [ 2088.973934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2090.016793] LNet: Removed LNI 192.168.206.58@tcp [ 2090.027801] LNet: Removed LNI 192.168.206.58@tcp2 [ 2090.062589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2090.123806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2090.148869] LNet: Added LNI 192.168.206.58@tcp [8/256/0/180] [ 2090.684446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2090.709896] LNet: Added LNI 192.168.206.58@tcp2 [8/256/0/180] [ 2091.480182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.58@tcp [ 2091.521664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2091.555143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2091.617920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.58@tcp [ 2091.937949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2093.024164] LNet: Removed LNI 192.168.206.58@tcp [ 2094.057895] LNet: Removed LNI 192.168.206.58@tcp2 [ 2094.104529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2094.186405] LNet: Added LNI 192.168.206.58@tcp [8/256/0/180] [ 2094.191726] LNet: Added LNI 192.168.206.58@tcp2 [8/256/0/180] [ 2094.699938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.58@tcp [ 2100.726347] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 2102.304557] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 15:48:40 (1789674520) [ 2102.383365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2102.424459] LNet: 119154:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2102.430070] LNet: 119154:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 2102.440323] LNet: Removed LNI 192.168.206.58@tcp [ 2103.463488] LNet: Removed LNI 192.168.206.58@tcp2 [ 2103.498485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2103.785263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2103.833942] LNet: Added LNI 192.168.206.58@tcp [8/256/0/180] [ 2104.456715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2104.495100] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2105.436666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2105.454651] LNetError: 119430:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2105.465190] LNetError: 119430:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 2105.488279] LNet: Removed LNI 10.1.2.3@tcp [ 2105.501633] LNet: Removed LNI 192.168.206.58@tcp [ 2105.526601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2105.603101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 2105.634305] LNet: Added LNI 192.168.206.58@tcp [8/256/0/180] [ 2105.638165] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2106.066785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 2106.093832] LNet: Removed LNI 192.168.206.58@tcp [ 2106.107614] LNet: Removed LNI 10.1.2.3@tcp [ 2111.599292] Lustre: DEBUG MARKER: == sanity-lnet test 475: per-operation latency stats populate and reset ========================================================== 15:48:49 (1789674529) [ 2111.766933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2111.809162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2111.861884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2111.877638] LNet: Added LNI 192.168.206.58@tcp [8/256/0/180] [ 2112.458956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2112.484650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2112.516494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.58@tcp [ 2112.567757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.58@tcp [ 2112.624715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.58@tcp [ 2112.663262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.58@tcp [ 2112.689880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.58@tcp [ 2112.724192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.58@tcp [ 2112.755609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.58@tcp [ 2112.793540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.58@tcp [ 2112.821163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.58@tcp [ 2112.849899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.58@tcp [ 2113.061175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2118.310685] Lustre: DEBUG MARKER: == sanity-lnet test 476: lnetctl set latency_stats toggles capture ========================================================== 15:48:56 (1789674536) [ 2118.456869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2119.531362] LNet: Removed LNI 192.168.206.58@tcp [ 2119.570780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2119.631814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2119.650664] LNet: Added LNI 192.168.206.58@tcp [8/256/0/180] [ 2120.252955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 0 [ 2120.296911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2120.356902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.58@tcp [ 2120.423898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.58@tcp [ 2120.449719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.58@tcp [ 2120.477435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.58@tcp [ 2120.507708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.58@tcp [ 2120.536277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.58@tcp [ 2120.564980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.58@tcp [ 2120.600970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.58@tcp [ 2120.635144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.58@tcp [ 2120.666547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.58@tcp [ 2120.743025] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2120.774306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.58@tcp [ 2120.802251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.58@tcp [ 2120.829677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.58@tcp [ 2120.864750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.58@tcp [ 2120.900612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.58@tcp [ 2120.923465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.58@tcp [ 2120.957909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.58@tcp [ 2120.996823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.58@tcp [ 2121.057434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.58@tcp [ 2121.100688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.58@tcp [ 2126.533064] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 15:49:04 (1789674544) [ 2126.733236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2127.785060] LNet: Removed LNI 192.168.206.58@tcp [ 2127.824711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2127.960939] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2128.109083] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2128.114253] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2128.118574] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2130.432929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2130.482364] LNet: Added LNI 192.168.206.58@tcp [8/256/0/180] [ 2131.325530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2131.363916] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2133.090371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.58@tcp [ 2133.128866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 2133.199047] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.206.58@tcp -m PUT -e local_timeout -r 1 [ 2133.261570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 2133.309866] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2133.314544] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2133.553943] LNetError: 121592:0:(lib-move.c:4794:lnet_parse()) 192.168.206.58@tcp, src 192.168.206.58@tcp: Dropping PUT (error -108 looking up sender) [ 2134.560569] LNet: 122051:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2134.574130] LNet: 122051:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 2134.594707] LNet: Removed LNI 10.1.2.3@tcp [ 2135.658319] LNet: Removed LNI 192.168.206.58@tcp [ 2136.050521] Key type .llcrypt unregistered [ 2136.052361] Key type ._llcrypt unregistered [ 2147.131946] Key type ._llcrypt registered [ 2147.134335] Key type .llcrypt registered [ 2148.417240] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 15:49:26 (1789674566) [ 2148.698598] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2148.714114] alg: No test for adler32 (adler32-zlib) [ 2149.610901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2149.983426] LNet: 122900:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2150.480469] Key type .llcrypt unregistered [ 2150.483740] Key type ._llcrypt unregistered [ 2156.207243] Key type ._llcrypt registered [ 2156.210739] Key type .llcrypt registered [ 2156.456471] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2156.481583] alg: No test for adler32 (adler32-zlib) [ 2157.396397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2157.424435] LNet: Added LNI 192.168.206.58@tcp [8/256/0/180] [ 2164.019717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.158@tcp [ 2164.211540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 2164.637587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.206.158@tcp [ 2164.858252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2164.895361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2164.941440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2164.965501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2164.992310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2165.032358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2165.066768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2165.086772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2165.127420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2165.153953] LNet: 123349:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2165.173933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2165.203521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2165.237176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2165.262276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2165.289739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2165.333581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2165.899768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2166.090674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2166.120471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2166.147531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2166.172928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2166.199157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2166.225324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2166.253397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2166.277880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2166.304653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2166.316756] LNet: 123469:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2166.329629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2166.353901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2166.381213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2166.404363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2166.426929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2166.463823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2167.008791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.206.158@tcp [ 2167.206438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2167.228649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2167.260873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2167.281459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2167.317417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2167.352863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2167.386511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2167.424956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2167.457949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2167.475495] LNet: 123588:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2167.509407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2167.557826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2167.603335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2167.630822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2167.672957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2167.708100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2168.242763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2168.454997] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2168.484909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2168.518089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2168.541188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2168.573288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2168.602679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2168.639921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2168.670741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2168.701535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2168.729705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2168.737116] LNet: 1 peer NIs in recovery (showing 1): 192.168.206.158@tcp [ 2168.761996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2168.799082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2168.833474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2168.858526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2168.888092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2169.186321] LNet: 123753:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2169.192353] LNetError: 123753:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2169.208923] LNet: Removed LNI 192.168.206.58@tcp [ 2169.576720] Key type .llcrypt unregistered [ 2169.578968] Key type ._llcrypt unregistered [ 2183.375691] Key type ._llcrypt registered [ 2183.378963] Key type .llcrypt registered [ 2184.538534] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 15:50:02 (1789674602) [ 2184.877132] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2184.900724] alg: No test for adler32 (adler32-zlib) [ 2185.761971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2186.100243] LNet: 124637:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2186.750422] Key type .llcrypt unregistered [ 2186.753084] Key type ._llcrypt unregistered [ 2192.165916] Key type ._llcrypt registered [ 2192.169853] Key type .llcrypt registered [ 2192.379649] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2192.402850] alg: No test for adler32 (adler32-zlib) [ 2193.305767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2193.332739] LNet: Added LNI 192.168.206.58@tcp [8/256/0/180] [ 2199.826553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.158@tcp [ 2207.986950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2208.032737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2208.100959] LNet: Added LNI 192.168.206.58@tcp1 [8/256/0/180] [ 2208.886848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 2209.421649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.206.158@tcp [ 2209.712344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2209.742097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2209.760114] LNet: 1 peer NIs in recovery (showing 1): 192.168.206.158@tcp [ 2209.810638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2209.880742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2209.918385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2209.959501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2210.011407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2210.049820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2210.104220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2210.120316] LNet: 125289:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2210.147587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2210.176411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2210.199470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2210.236111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2210.287168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2210.316695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2210.939442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2211.202385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2211.228138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2211.259886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2211.301334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2211.333617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2211.364357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2211.438699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2211.489227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2211.537042] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2211.573717] LNet: 125408:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2211.593459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2211.625310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2211.679113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2211.718229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2211.744751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2211.775539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2212.305777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.206.158@tcp [ 2212.522679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2212.550681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2212.585284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2212.616243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2212.656777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2212.699539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2212.738685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2212.764369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2212.802678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2212.817974] LNet: 125528:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2212.836787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2212.863715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2212.897502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2212.948463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2212.988378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2213.031805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2213.619127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2213.865283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2213.894607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2213.924377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2213.951285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2213.984468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2214.019477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2214.054109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2214.084679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2214.115857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2214.150537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2214.191608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2214.227043] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2214.256794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2214.286605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2214.315765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2218.827538] LNet: 125728:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2218.837615] LNetError: 125728:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2218.852146] LNet: Removed LNI 192.168.206.58@tcp [ 2218.867557] LNet: Removed LNI 192.168.206.58@tcp1 [ 2219.291816] Key type .llcrypt unregistered [ 2219.296496] Key type ._llcrypt unregistered [ 2232.849404] Key type ._llcrypt registered [ 2232.851978] Key type .llcrypt registered [ 2233.955985] Lustre: DEBUG MARKER: == sanity-lnet test 525: keep reachable peer on local net delete ========================================================== 15:50:52 (1789674652) [ 2234.287963] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2234.310814] alg: No test for adler32 (adler32-zlib) [ 2235.170535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2235.217587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2235.238284] LNet: Added LNI 192.168.206.58@tcp [8/256/0/180] [ 2235.990414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2236.034458] LNet: Added LNI 192.168.206.58@tcp2 [8/256/0/180] [ 2236.855954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.58@tcp [ 2236.922938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp [ 2237.984178] LNet: Removed LNI 192.168.206.58@tcp [ 2244.232500] Lustre: DEBUG MARKER: == sanity-lnet test 550: DD-off: keep route up when gw reachable via another local NI ========================================================== 15:51:02 (1789674662) [ 2245.517505] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_550 Need at least 2 local interfaces for multi-rail [ 2247.193691] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 15:51:05 (1789674665) [ 2253.285623] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 15:51:11 (1789674671) [ 2259.798238] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 15:51:18 (1789674678) [ 2265.924806] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 15:51:24 (1789674684) [ 2272.052218] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 15:51:30 (1789674690) [ 2278.052683] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 15:51:36 (1789674696) [ 2283.924318] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 15:51:42 (1789674702) [ 2290.885830] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 15:51:48 (1789674708) [ 2299.140144] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 15:51:56 (1789674716) [ 2306.437458] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 15:52:04 (1789674724) [ 2312.935832] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 15:52:11 (1789674731) [ 2319.713509] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 15:52:17 (1789674737) [ 2325.116039] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 15:52:23 (1789674743) [ 2330.431364] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 15:52:28 (1789674748) [ 2336.206585] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 15:52:34 (1789674754) [ 2343.751922] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 15:52:42 (1789674762) [ 2351.589594] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 15:52:49 (1789674769) [ 2357.713686] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 15:52:56 (1789674776) [ 2357.780697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2357.820533] LNet: 137411:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2357.825847] LNetError: 137411:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2357.840706] LNet: Removed LNI 192.168.206.58@tcp2 [ 2357.869829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2357.977355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2357.995269] LNet: Added LNI 192.168.206.58@tcp [8/256/0/180] [ 2358.981992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2359.012082] LNet: 137559:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2359.021716] LNetError: 137559:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2359.034716] LNet: Removed LNI 192.168.206.58@tcp [ 2359.063480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2359.120818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2359.142690] LNet: Added LNI 192.168.206.58@tcp [8/256/0/180] [ 2365.333250] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 15:53:03 (1789674783) [ 2371.920712] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 15:53:10 (1789674790) [ 2378.146894] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 15:53:16 (1789674796) [ 2385.249045] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 15:53:23 (1789674803) [ 2391.507793] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 15:53:29 (1789674809) [ 2397.567864] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 15:53:35 (1789674815) [ 2403.524573] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 15:53:41 (1789674821) [ 2410.270683] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 15:53:48 (1789674828) [ 2416.590874] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 15:53:54 (1789674834) [ 2423.089779] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 15:54:01 (1789674841) [ 2430.182387] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 15:54:08 (1789674848) [ 2437.364285] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 15:54:15 (1789674855) [ 2443.647479] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 2184 sec ========== 15:54:21 (1789674861) [ 2444.014140] LNet: 145024:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2444.018733] LNetError: 145024:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2444.032349] LNet: Removed LNI 192.168.206.58@tcp [ 2444.596827] Key type .llcrypt unregistered [ 2444.599232] Key type ._llcrypt unregistered [ 2455.256088] Key type ._llcrypt registered [ 2455.264726] Key type .llcrypt registered [ 2455.804295] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2455.820546] alg: No test for adler32 (adler32-zlib) [ 2456.976905] Lustre: Lustre: Build Version: 2.17.58_39_g9373f0b [ 2457.191060] LNet: Added LNI 192.168.206.58@tcp [8/256/0/180] [ 2458.888326] Key type lgssc registered [ 2460.050287] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2519.663116] Lustre: Mounted lustre-client - version 2.17.58_39_g9373f0b [ 2524.428511] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2529.871384] Lustre: Unmounted lustre-client [ 2549.751356] Key type lgssc unregistered [ 2549.971324] LNet: 147970:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2549.988783] LNetError: 147970:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2550.014028] LNet: Removed LNI 192.168.206.58@tcp [ 2550.683675] Key type .llcrypt unregistered [ 2550.687122] Key type ._llcrypt unregistered