[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 483100534 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895288K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001010] APIC: Switch to symmetric I/O mode setup [ 0.003268] x2apic enabled [ 0.004008] Switched APIC routing to physical x2apic. [ 0.005011] kvm-guest: setup PV IPIs [ 0.008453] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009021] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010013] pid_max: default: 32768 minimum: 301 [ 0.011150] LSM: Security Framework initializing [ 0.012055] Yama: becoming mindful. [ 0.013052] SELinux: Initializing. [ 0.014074] *** VALIDATE selinux *** [ 0.022546] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026751] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027147] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028114] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029113] *** VALIDATE tmpfs *** [ 0.031248] *** VALIDATE proc *** [ 0.032246] *** VALIDATE cgroup *** [ 0.033010] *** VALIDATE cgroup2 *** [ 0.034267] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.035158] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.036010] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.037029] Spectre V2 : User space: Vulnerable [ 0.038008] Speculative Store Bypass: Vulnerable [ 0.041203] debug: unmapping init [mem 0xffffffff8d459000-0xffffffff8d460fff] [ 0.044111] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045712] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046034] ... version: 2 [ 0.047013] ... bit width: 48 [ 0.048011] ... generic registers: 4 [ 0.049014] ... value mask: 0000ffffffffffff [ 0.050015] ... max period: 00007fffffffffff [ 0.051016] ... fixed-purpose events: 3 [ 0.052011] ... event mask: 000000070000000f [ 0.053275] rcu: Hierarchical SRCU implementation. [ 0.055465] smp: Bringing up secondary CPUs ... [ 0.056590] x86: Booting SMP configuration: [ 0.057039] .... node #0, CPUs: #1 #2 #3 [ 0.061075] smp: Brought up 1 node, 4 CPUs [ 0.063018] smpboot: Max logical packages: 1 [ 0.064018] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.135894] node 0 deferred pages initialised in 68ms [ 0.142101] devtmpfs: initialized [ 0.143277] x86/mm: Memory block size: 128MB [ 0.146741] gcov: version magic: 0x41383552 [ 0.149420] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.150081] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.151242] pinctrl core: initialized pinctrl subsystem [ 0.152185] [ 0.152599] ************************************************************* [ 0.153014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.154015] ** ** [ 0.155026] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.156013] ** ** [ 0.157012] ** This means that this kernel is built to expose internal ** [ 0.158009] ** IOMMU data structures, which may compromise security on ** [ 0.159014] ** your system. ** [ 0.160013] ** ** [ 0.161011] ** If you see this message and you are not debugging the ** [ 0.162013] ** kernel, report this immediately to your vendor! ** [ 0.163014] ** ** [ 0.164015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.165013] ************************************************************* [ 0.166827] NET: Registered protocol family 16 [ 0.168570] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.172066] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.174126] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.178099] cpuidle: using governor menu [ 0.180840] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.183562] PCI: Using configuration type 1 for base access [ 0.185183] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.194120] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.195043] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.196149] cryptd: max_cpu_qlen set to 1000 [ 0.199221] ACPI: Added _OSI(Module Device) [ 0.201011] ACPI: Added _OSI(Processor Device) [ 0.202013] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.204014] ACPI: Added _OSI(Processor Aggregator Device) [ 0.209298] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.215480] ACPI: Interpreter enabled [ 0.217057] ACPI: PM: (supports S0 S3 S4 S5) [ 0.219013] ACPI: Using IOAPIC for interrupt routing [ 0.220112] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.223515] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.233834] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.236060] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.239025] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.243190] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.249599] acpiphp: Slot [2] registered [ 0.251103] acpiphp: Slot [5] registered [ 0.252097] acpiphp: Slot [6] registered [ 0.253111] acpiphp: Slot [3] registered [ 0.254083] acpiphp: Slot [4] registered [ 0.255078] acpiphp: Slot [7] registered [ 0.257093] acpiphp: Slot [8] registered [ 0.258124] acpiphp: Slot [9] registered [ 0.259072] acpiphp: Slot [10] registered [ 0.260096] acpiphp: Slot [11] registered [ 0.261080] acpiphp: Slot [12] registered [ 0.262130] acpiphp: Slot [13] registered [ 0.264107] acpiphp: Slot [14] registered [ 0.265155] acpiphp: Slot [15] registered [ 0.267095] acpiphp: Slot [16] registered [ 0.269097] acpiphp: Slot [17] registered [ 0.270095] acpiphp: Slot [18] registered [ 0.272093] acpiphp: Slot [19] registered [ 0.275104] acpiphp: Slot [20] registered [ 0.276114] acpiphp: Slot [21] registered [ 0.278105] acpiphp: Slot [22] registered [ 0.280134] acpiphp: Slot [23] registered [ 0.281085] acpiphp: Slot [24] registered [ 0.282097] acpiphp: Slot [25] registered [ 0.284109] acpiphp: Slot [26] registered [ 0.285093] acpiphp: Slot [27] registered [ 0.287095] acpiphp: Slot [28] registered [ 0.288122] acpiphp: Slot [29] registered [ 0.290092] acpiphp: Slot [30] registered [ 0.291094] acpiphp: Slot [31] registered [ 0.292049] PCI host bridge to bus 0000:00 [ 0.293017] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.295018] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.298029] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.301028] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.303021] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.306045] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.308169] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.311027] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.315256] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.322014] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.327080] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.330024] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.332016] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.335016] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.338489] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.340724] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.343040] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.345724] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.349947] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.360014] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.365014] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.370000] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.378014] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.382027] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.394017] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.401427] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.408014] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.414015] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.428015] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.438605] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.440336] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.442275] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.444283] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.445202] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.450106] iommu: Default domain type: Passthrough [ 0.452340] SCSI subsystem initialized [ 0.454109] ACPI: bus type USB registered [ 0.455096] usbcore: registered new interface driver usbfs [ 0.457112] usbcore: registered new interface driver hub [ 0.458081] usbcore: registered new device driver usb [ 0.460125] pps_core: LinuxPPS API ver. 1 registered [ 0.462013] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.464056] PTP clock support registered [ 0.466119] EDAC MC: Ver: 3.0.0 [ 0.467347] PCI: Using ACPI for IRQ routing [ 0.468963] NetLabel: Initializing [ 0.469013] NetLabel: domain hash size = 128 [ 0.470010] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.471087] NetLabel: unlabeled traffic allowed by default [ 0.472126] vgaarb: loaded [ 0.474259] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.475011] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.479454] clocksource: Switched to clocksource kvm-clock [ 0.582055] VFS: Disk quotas dquot_6.6.0 [ 0.583617] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.586415] *** VALIDATE ramfs *** [ 0.587603] *** VALIDATE hugetlbfs *** [ 0.589086] pnp: PnP ACPI init [ 0.591228] pnp: PnP ACPI: found 6 devices [ 0.608076] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.611764] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.614225] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.616589] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.619737] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.622765] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.626091] NET: Registered protocol family 2 [ 0.628642] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.633584] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.637957] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.643283] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.646905] TCP: Hash tables configured (established 65536 bind 65536) [ 0.649946] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.653233] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.657387] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.660636] NET: Registered protocol family 1 [ 0.663141] RPC: Registered named UNIX socket transport module. [ 0.664974] RPC: Registered udp transport module. [ 0.666704] RPC: Registered tcp transport module. [ 0.668539] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.670904] NET: Registered protocol family 44 [ 0.672736] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.675149] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.677442] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.680045] PCI: CLS 0 bytes, default 64 [ 0.682207] Unpacking initramfs... [ 2.062486] debug: unmapping init [mem 0xffff94587cc64000-0xffff94587ffcffff] [ 2.066576] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.068717] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.071725] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.571959] Initialise system trusted keyrings [ 2.573816] Key type blacklist registered [ 2.575830] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.586107] zbud: loaded [ 2.589174] *** VALIDATE nfs *** [ 2.590561] *** VALIDATE nfs4 *** [ 2.592166] pstore: using deflate compression [ 2.596366] Platform Keyring initialized [ 2.696155] NET: Registered protocol family 38 [ 2.697357] Key type asymmetric registered [ 2.698544] Asymmetric key parser 'x509' registered [ 2.699933] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.702378] io scheduler mq-deadline registered [ 2.703943] io scheduler kyber registered [ 2.705752] io scheduler bfq registered [ 2.708078] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.710429] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.713350] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.716303] ACPI: Power Button [PWRF] [ 2.721973] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.728095] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.739539] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.769740] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.798947] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.803969] Non-volatile memory driver v1.3 [ 2.805574] Linux agpgart interface v0.103 [ 2.839838] virtio_blk virtio1: [vda] 146992 512-byte logical blocks (75.3 MB/71.8 MiB) [ 2.843344] vda: detected capacity change from 0 to 75259904 [ 2.860655] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 2.864223] vdb: detected capacity change from 0 to 1073741824 [ 2.880261] libphy: Fixed MDIO Bus: probed [ 2.891171] usbcore: registered new interface driver usbserial_generic [ 2.893468] usbserial: USB Serial support registered for generic [ 2.895566] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 2.899738] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 2.901501] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 2.903305] mousedev: PS/2 mouse device common for all mice [ 2.905518] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 2.908460] rtc_cmos 00:05: RTC can wake from S4 [ 2.912840] rtc_cmos 00:05: registered as rtc0 [ 2.913302] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 2.915311] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 2.922256] intel_pstate: CPU model not supported [ 2.924332] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 2.929584] hid: raw HID events driver (C) Jiri Kosina [ 2.932088] usbcore: registered new interface driver usbhid [ 2.934327] usbhid: USB HID core driver [ 2.935351] drop_monitor: Initializing network drop monitor service [ 2.937545] Initializing XFRM netlink socket [ 2.939489] NET: Registered protocol family 10 [ 2.942840] Segment Routing with IPv6 [ 2.944403] NET: Registered protocol family 17 [ 2.946623] mpls_gso: MPLS GSO support [ 2.951051] RAS: Correctable Errors collector initialized. [ 2.953094] AVX version of gcm_enc/dec engaged. [ 2.954785] AES CTR mode by8 optimization enabled [ 3.024409] sched_clock: Marking stable (3024340922, 0)->(3965358346, -941017424) [ 3.028426] registered taskstats version 1 [ 3.030439] Loading compiled-in X.509 certificates [ 3.032850] zswap: loaded using pool lzo/zbud [ 3.060719] Key type big_key registered [ 3.071589] Key type encrypted registered [ 3.072905] ima: No TPM chip found, activating TPM-bypass! [ 3.074228] ima: Allocated hash algorithm: sha1 [ 3.075392] ima: No architecture policies found [ 3.076751] evm: Initialising EVM extended attributes: [ 3.078071] evm: security.selinux [ 3.078824] evm: security.ima [ 3.079588] evm: security.capability [ 3.080491] evm: HMAC attrs: 0x1 [ 3.084183] rtc_cmos 00:05: setting system clock to 2026-08-27 20:39:25 UTC (1787863165) [ 3.090722] debug: unmapping init [mem 0xffffffff8e403000-0xffffffff8e5fffff] [ 3.092988] debug: unmapping init [mem 0xffffffff8d182000-0xffffffff8d458fff] [ 3.100103] Write protecting the kernel read-only data: 28672k [ 3.103158] debug: unmapping init [mem 0xffffffff8b803000-0xffffffff8b9fffff] [ 3.105381] debug: unmapping init [mem 0xffffffff8c114000-0xffffffff8c1fffff] [ 3.130376] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.136707] systemd[1]: Detected virtualization kvm. [ 3.138012] systemd[1]: Detected architecture x86-64. [ 3.139226] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.164839] systemd[1]: No hostname configured. [ 3.166271] systemd[1]: Set hostname to . [ 3.167696] random: systemd: uninitialized urandom read (16 bytes read) [ 3.169321] systemd[1]: Initializing machine ID from random generator. [ 3.293795] random: systemd: uninitialized urandom read (16 bytes read) [ 3.297066] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 3.301903] random: systemd: uninitialized urandom read (16 bytes read) [ 3.303927] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 3.308302] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Swap. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Timers. [ OK ] Listening on Journal Socket. Starting Journal Service... Starting Create Volatile Files and Directories... Starting Setup Virtual Console... [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Sockets. Starting Apply Kernel Variables... [ OK ] Reached target Slices. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Paths. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Journal Service. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 3.941478] device-mapper: uevent: version 1.0.3 [ 3.943816] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 4.579321] virtio_net virtio0 ens2: renamed from eth0 [ 4.604372] random: fast init done [ 4.658718] scsi host0: ata_piix [ 4.685740] scsi host1: ata_piix [ 4.755558] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 4.757496] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 8.274260] dracut-initqueue[576]: RTNETLINK answers: File exists [ 9.588721] random: crng init done [ 9.590285] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 9.857121] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 10.953817] printk: systemd: 22 output lines suppressed due to ratelimiting [ 11.218338] SELinux: Disabled at runtime. [ 11.280201] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.289459] systemd[1]: Detected virtualization kvm. [ 11.291279] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 11.751400] systemd[1]: initrd-switch-root.service: Succeeded. [ 11.754317] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 11.759892] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 11.763108] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 11.765572] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 11.772939] systemd[1]: Starting Journal Service... Starting Journal Service... [ 11.777295] systemd[1]: Listening on initctl Compatibility Named Pipe. [ OK ] Listening on initctl Compatibility Named Pipe. Mounting Huge Pages File System... Starting Create list of required st…ce nodes for the current kernel... Activating swap /dev/disk/by-label/SWAP... Starting Apply Kernel Variables... [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on udev Control Socket. [ OK ] Created slice system-getty.slice. [ 11.823773] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Mounting Kernel Debug File System... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice system-serial\x2dgetty.slice. Mounting POSIX Message Queue File System... [ OK ] Listening on Process Core Dump Socket. [ OK ] Stopped target Switch Root. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Starting Remount Root and Kernel File Systems... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Started Journal Service. [ OK ] Mounted Huge Pages File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted POSIX Message Queue File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ OK ] Started udev Coldplug all Devices. [ 12.209969] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 12.492466] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 12.606211] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 12.666611] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 12.682176] EDAC sbridge: Ver: 1.1.2 [ 13.780479] Key type dns_resolver registered [ 14.111885] NFS: Registering the id_resolver key type [ 14.114369] Key type id_resolver registered [ 14.115692] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Basic System. Starting Login Service... Starting Restore /run/initramfs on shutdown... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started irqbalance daemon. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... [ OK ] Started OpenSSH server daemon. Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg630-client login: [ 40.467442] hrtimer: interrupt took 3594936 ns [ 73.321946] libcfs: loading out-of-tree module taints kernel. [ 73.410110] Key type ._llcrypt registered [ 73.420085] Key type .llcrypt registered [ 74.148606] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 74.164824] alg: No test for adler32 (adler32-zlib) [ 75.732525] Lustre: Lustre: Build Version: 2.17.57_106_g3758560 [ 76.824492] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 78.695161] Key type lgssc registered [ 80.768768] Lustre: Echo OBD driver; http://www.lustre.org/ [ 211.547771] Lustre: Mounted lustre-client - version 2.17.57_106_g3758560 [ 216.556898] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 229.937577] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing check_logdir /tmp/testlogs/ [ 235.273300] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing yml_node [ 237.024141] Lustre: lustre-OST0000-osc-ffff9458c7aea800: disconnect after 23s idle [ 242.134273] Lustre: DEBUG MARKER: Client: 2.17.57.106 [ 244.446511] Lustre: DEBUG MARKER: MDS: 2.17.57.106 [ 247.188225] Lustre: DEBUG MARKER: OSS: 2.17.57.106 [ 248.661105] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Thu Aug 27 16:43:29 EDT 2026 [ 264.858884] Lustre: Unmounted lustre-client [ 303.843808] Key type lgssc unregistered [ 304.139868] LNet: 8413:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 304.150637] LNetError: 8413:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 305.194647] LNet: Removed LNI 192.168.206.30@tcp [ 305.952328] Key type .llcrypt unregistered [ 305.956171] Key type ._llcrypt unregistered [ 333.536797] Key type ._llcrypt registered [ 333.541635] Key type .llcrypt registered [ 333.722593] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 333.746829] alg: No test for adler32 (adler32-zlib) [ 334.693529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 334.763035] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 334.777478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 339.236143] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 339.397933] LNet: 9300:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 339.423221] LNetError: 9300:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 339.452064] LNet: Removed LNI 192.168.206.30@tcp [ 340.226470] Key type .llcrypt unregistered [ 340.232600] Key type ._llcrypt unregistered [ 347.915965] Key type ._llcrypt registered [ 347.921343] Key type .llcrypt registered [ 349.444431] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 16:45:10 (1787863510) [ 349.880415] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 349.912594] alg: No test for adler32 (adler32-zlib) [ 350.794681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 350.869533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 350.890679] LNet: 9720:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 350.913707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 350.980617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 351.010359] LNet: 9727:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 358.319490] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 16:45:19 (1787863519) [ 358.606647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 358.637809] LNet: 10305:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 358.655914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 358.771828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 366.854834] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 16:45:27 (1787863527) [ 367.217383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 367.262210] LNet: 10892:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 367.304746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 367.426344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 374.826404] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 16:45:35 (1787863535) [ 375.117825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 375.140957] LNet: 11477:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 375.167393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 375.252174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 381.538711] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 16:45:42 (1787863542) [ 381.839545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 381.867897] LNet: 12063:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 381.904345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 382.020468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 382.529083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 388.980729] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 16:45:50 (1787863550) [ 389.150544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 389.206973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 389.307846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 390.089446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 390.974631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 397.449888] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 16:45:58 (1787863558) [ 397.589813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 397.618932] LNet: 13246:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 397.636540] LNet: 13246:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 397.665235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 397.767794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 404.288273] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 16:46:05 (1787863565) [ 404.521944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 404.560922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 404.650904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 404.696061] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 405.310703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 405.373821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 405.942735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 405.991396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 406.573408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 406.620761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 407.177223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 407.227898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 407.778911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 407.831577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 408.240048] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 408.300832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 408.624197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 408.661156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 409.085715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 409.132928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 409.564735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 409.624922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 410.108829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 410.150163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 410.609433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 410.643444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 410.960852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 411.006818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 417.451932] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 16:46:18 (1787863578) [ 417.735373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 417.775673] LNet: 14516:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 417.781801] LNet: 14516:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 417.808902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 417.897367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 417.949790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 425.027470] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 16:46:25 (1787863585) [ 425.119121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 425.194966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 425.318786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 425.370470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 432.295603] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 16:46:33 (1787863593) [ 432.605525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 432.661432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 432.756865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 432.794216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 439.382402] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 16:46:40 (1787863600) [ 439.458660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 439.528462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 439.686137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 439.734342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 445.772862] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 16:46:47 (1787863607) [ 446.176278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 446.228390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 446.325856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 446.370142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 452.716856] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 16:46:53 (1787863613) [ 453.085455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 453.109416] LNet: 17447:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 453.115240] LNet: 17447:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 453.135647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 453.194199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 453.234947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 459.737318] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 16:47:00 (1787863620) [ 459.858230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 459.901166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 460.031531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 460.538308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 467.448556] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 16:47:08 (1787863628) [ 467.634799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 467.689951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 467.914839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 468.179832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 474.639630] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 16:47:16 (1787863636) [ 474.964745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 474.996523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 475.126464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 475.348528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 481.898423] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 16:47:22 (1787863642) [ 482.428218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 482.463558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 483.176613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 483.382395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 489.760852] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 16:47:30 (1787863650) [ 490.154394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 490.226729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 490.340451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 490.401292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 497.291870] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 16:47:38 (1787863658) [ 497.435533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 497.500692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 497.590108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 497.635828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 504.440611] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 16:47:45 (1787863665) [ 504.800909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 504.848675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 504.956888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 513.418607] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 16:47:54 (1787863674) [ 513.640108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 513.695828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 513.792189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 520.981751] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 16:48:02 (1787863682) [ 521.252379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 521.289320] LNet: 22883:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 521.297212] LNet: 22883:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 8 previous similar messages [ 521.327930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 521.423401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 522.036616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 529.902629] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 16:48:10 (1787863690) [ 530.303619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 530.359962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 530.435981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 530.467595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 536.922052] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 16:48:18 (1787863698) [ 537.210986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 537.280418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 537.348271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 537.376421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 543.784918] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 16:48:25 (1787863705) [ 543.883248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 543.961176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 544.064232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 544.094520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 550.375890] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 16:48:31 (1787863711) [ 550.533092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 550.574785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 550.706857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 550.746153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 557.782076] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 16:48:38 (1787863718) [ 557.913682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 557.987247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 558.077523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 558.203972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 558.222874] LNetError: 25831:0:(peer.c:2362:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 558.264545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 558.293995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 564.887471] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 16:48:45 (1787863725) [ 565.059317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 565.158097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 565.247346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.VZfF9yxzrq [ 565.349759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.VZfF9yxzrq [ 565.472843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.VZfF9yxzrq [ 571.734081] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 16:48:52 (1787863732) [ 572.088754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 572.126762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 572.178230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 572.219886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 572.280561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 579.248688] Lustre: DEBUG MARKER: == sanity-lnet test 40: LNetAddPeer merges a pre-existing non-primary peer NID ========================================================== 16:49:00 (1787863740) [ 579.657057] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 579.704336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 579.774955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 579.806542] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 581.503185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.30@tcp [ 583.206699] Lustre: Lustre: Build Version: 2.17.57_106_g3758560 [ 585.351381] Key type lgssc registered [ 586.878902] Lustre: Echo OBD driver; http://www.lustre.org/ [ 590.829262] LNetError: 27592:0:(lib-move.c:2158:lnet_handle_find_routed_path()) peer 192.168.206.30@tcp99 has no available nets [ 593.442604] Key type lgssc unregistered [ 593.858417] LNetError: 28704:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 593.879328] LNet: Removed LNI 192.168.206.30@tcp [ 594.515304] Key type .llcrypt unregistered [ 594.517979] Key type ._llcrypt unregistered [ 601.235478] Key type ._llcrypt registered [ 601.238813] Key type .llcrypt registered [ 602.447814] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 16:49:23 (1787863763) [ 602.984935] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 603.011372] alg: No test for adler32 (adler32-zlib) [ 603.915023] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 604.015872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 606.607844] LNet: Message forwarding will be enabled in 2s [ 609.695488] LNet: Message forwarding enabled [ 610.049711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 610.066932] LNet: Message forwarding will stop in 2s to 6s [ 614.738103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 615.591394] LNet: Message forwarding disabled [ 617.389685] LNet: Message forwarding will be enabled in 2s [ 620.715349] LNet: Message forwarding enabled [ 624.436650] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 16:49:45 (1787863785) [ 624.518877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 624.554108] LNet: 30105:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 625.792234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 625.856536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 632.208712] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 16:49:53 (1787863793) [ 632.562750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 632.595827] LNet: 30695:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 632.645673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 632.700741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 632.739608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 632.778985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 632.833825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 632.864800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 632.900370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 632.932616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 632.980249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 633.018318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 633.047908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 633.081023] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 633.115520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 633.150547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 633.186686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 633.224183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 633.257673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 633.294940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 633.336901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 633.371709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 633.386507] LNetError: 30738:0:(router.c:625:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 639.358384] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 16:50:00 (1787863800) [ 639.458707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 639.481912] LNet: 31304:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 639.499425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 639.571563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 645.917909] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 16:50:07 (1787863807) [ 647.783337] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg630-server [ 649.571703] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 16:50:10 (1787863810) [ 649.678333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 649.726462] LNet: 32233:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 649.757893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 649.844396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 649.875776] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 650.545513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.206.[31-35]@tcp [ 650.568907] LNet: 32373:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 650.617429] LNetError: 32244:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.206.31@tcp is being used as a gateway but routing feature is not turned on [ 650.636098] LNetError: 32244:0:(lib-lnet.h:1280:lnet_set_route_aliveness()) route to tcp1 through 192.168.206.31@tcp has gone from up to down [ 651.636501] LNetError: 32244:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.206.34@tcp is being used as a gateway but routing feature is not turned on [ 651.651900] LNetError: 32244:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 652.651506] LNetError: 32244:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.206.32@tcp is being used as a gateway but routing feature is not turned on [ 652.667456] LNetError: 32244:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 4 previous similar messages [ 654.695791] LNetError: 32244:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.206.34@tcp is being used as a gateway but routing feature is not turned on [ 654.708160] LNetError: 32244:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 656.656916] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 16:50:18 (1787863818) [ 658.721500] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg630-server [ 658.799664] LNetError: 32244:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.206.33@tcp is being used as a gateway but routing feature is not turned on [ 658.819981] LNetError: 32244:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 19 previous similar messages [ 660.475401] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 16:50:21 (1787863821) [ 660.557812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 660.609743] LNet: 33304:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 660.620741] LNetError: 33304:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 660.666691] LNet: Removed LNI 192.168.206.30@tcp [ 660.713575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 660.763617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 660.809116] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 661.375463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.206.[31-36/2]@tcp [ 661.393606] LNet: 33427:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 661.401969] LNet: 33427:0:(router.c:729:lnet_add_route()) Skipped 4 previous similar messages [ 661.419111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.206.[31-36/2]@tcp [ 661.426525] LNetError: 33314:0:(lib-lnet.h:1280:lnet_set_route_aliveness()) route to tcp103 through 192.168.206.35@tcp has gone from up to down [ 661.451375] LNetError: 33314:0:(lib-lnet.h:1280:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 668.241180] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 16:50:29 (1787863829) [ 668.527553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 668.547898] LNetError: 34021:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 668.571811] LNet: Removed LNI 192.168.206.30@tcp [ 668.583953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 668.668048] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 668.750420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 668.779556] LNet: 34035:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 668.786868] LNet: 34035:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 668.817583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 668.888903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 668.947868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 668.995546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 669.063360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 669.140942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 669.206735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 669.288808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 669.375207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 669.410978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 669.467725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 669.541828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 669.568380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 669.641791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 669.711435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 669.752756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 669.804525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 669.884719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 669.930464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 669.996459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 670.085764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 670.132990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 670.192887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 670.300995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 670.338118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 670.455360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 670.537647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 670.597746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 670.692031] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 670.766418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 670.829921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 670.895643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 676.775779] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 16:50:37 (1787863837) [ 678.783978] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg630-server [ 680.644927] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 16:50:41 (1787863841) [ 682.741700] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg630-server [ 684.529668] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 16:50:45 (1787863845) [ 685.456542] Key type .llcrypt unregistered [ 685.457744] Key type ._llcrypt unregistered [ 692.659492] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 692.892534] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 692.905406] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 692.910885] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 695.449833] Key type ._llcrypt registered [ 695.454288] Key type .llcrypt registered [ 695.615542] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 695.628778] alg: No test for adler32 (adler32-zlib) [ 696.555859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 696.628861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 696.651591] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 697.436335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 697.455352] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 699.448678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 699.481679] LNet: Removed LNI 10.1.2.3@tcp [ 699.619195] LNet: 36148:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 699.627817] LNetError: 36148:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 699.654958] LNet: Removed LNI 192.168.206.30@tcp [ 700.485392] Key type .llcrypt unregistered [ 700.488311] Key type ._llcrypt unregistered [ 714.358230] Key type ._llcrypt registered [ 714.360345] Key type .llcrypt registered [ 716.278500] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 16:51:16 (1787863876) [ 717.101443] Key type .llcrypt unregistered [ 717.104691] Key type ._llcrypt unregistered [ 724.976148] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 725.186453] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 725.198891] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 725.211159] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 727.973453] Key type ._llcrypt registered [ 727.976308] Key type .llcrypt registered [ 728.116192] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 728.125160] alg: No test for adler32 (adler32-zlib) [ 728.979270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 729.025744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 729.043821] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 730.130582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 730.167102] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 732.994478] LNet: 37650:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 733.005459] LNetError: 37650:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 733.013200] LNet: Removed LNI 10.1.2.3@tcp [ 733.027080] LNet: Removed LNI 192.168.206.30@tcp [ 733.820405] Key type .llcrypt unregistered [ 733.823055] Key type ._llcrypt unregistered [ 745.702633] Key type ._llcrypt registered [ 745.703727] Key type .llcrypt registered [ 746.947592] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 16:51:48 (1787863908) [ 747.585325] Key type .llcrypt unregistered [ 747.586985] Key type ._llcrypt unregistered [ 753.578200] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 753.739266] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 753.741939] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 753.744725] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 756.218694] Key type ._llcrypt registered [ 756.222031] Key type .llcrypt registered [ 756.456963] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 756.479034] alg: No test for adler32 (adler32-zlib) [ 757.463179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 757.530409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 757.570964] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 759.764308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 759.801218] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 761.805738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 761.834356] LNet: Removed LNI 10.1.2.3@tcp [ 761.861366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 761.932095] LNet: Removed LNI 10.1.2.31@tcp [ 761.968412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 762.001990] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 762.869625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 762.893144] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 763.916582] LNet: 39579:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 763.924818] LNetError: 39579:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg:0 not found [ 763.942156] LNet: Removed LNI 10.1.2.31@tcp [ 763.963560] LNet: Removed LNI 10.1.2.3@tcp [ 764.835650] Key type .llcrypt unregistered [ 764.837458] Key type ._llcrypt unregistered [ 779.044489] Key type ._llcrypt registered [ 779.067029] Key type .llcrypt registered [ 780.778552] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 16:52:21 (1787863941) [ 781.266836] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 781.289112] alg: No test for adler32 (adler32-zlib) [ 782.264867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 782.317515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 782.348165] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 783.208348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 783.243159] LNet: 40537:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 783.257870] LNetError: 40537:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 783.294293] LNet: Removed LNI 192.168.206.30@tcp [ 783.349719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 783.439454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.206.30@tcp [ 783.484761] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 784.655681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.206.30@tcp [ 784.695353] LNet: Removed LNI 192.168.206.30@tcp [ 784.806279] LNet: 40672:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 785.795578] Key type .llcrypt unregistered [ 785.798257] Key type ._llcrypt unregistered [ 797.974553] Key type ._llcrypt registered [ 797.976357] Key type .llcrypt registered [ 799.315043] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 16:52:40 (1787863960) [ 801.519407] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg630-server [ 803.653929] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 16:52:44 (1787863964) [ 804.273308] Key type .llcrypt unregistered [ 804.275713] Key type ._llcrypt unregistered [ 810.579096] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 810.791783] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 810.801388] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 810.809266] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 813.309329] Key type ._llcrypt registered [ 813.310754] Key type .llcrypt registered [ 813.559274] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 813.575604] alg: No test for adler32 (adler32-zlib) [ 814.440376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 814.536385] LNet: Added LNI 192.168.206.30@tcp1 [8/256/0/180] [ 815.228327] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 817.592959] LNet: 42523:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 817.599783] LNetError: 42523:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 817.604254] LNet: Removed LNI 192.168.206.30@tcp1 [ 817.612976] LNet: Removed LNI 10.1.2.3@tcp2 [ 818.364304] Key type .llcrypt unregistered [ 818.371121] Key type ._llcrypt unregistered [ 830.664681] Key type ._llcrypt registered [ 830.672114] Key type .llcrypt registered [ 832.313140] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 16:53:12 (1787863992) [ 832.729634] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 832.743449] alg: No test for adler32 (adler32-zlib) [ 833.631597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 833.685712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 833.705657] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 834.530899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 834.553414] LNet: 43497:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 834.559787] LNetError: 43497:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 834.572889] LNet: Removed LNI 192.168.206.30@tcp [ 834.597280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 834.665681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 834.712757] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 841.681565] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 16:53:22 (1787864002) [ 841.859045] LNet: 44081:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 841.883486] LNetError: 44081:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 841.922439] LNet: Removed LNI 192.168.206.30@tcp [ 842.791744] Key type .llcrypt unregistered [ 842.794081] Key type ._llcrypt unregistered [ 849.154703] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 849.251470] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 849.254906] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 849.258503] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 852.004557] Key type ._llcrypt registered [ 852.008480] Key type .llcrypt registered [ 852.181800] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 852.196580] alg: No test for adler32 (adler32-zlib) [ 853.171955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 853.249654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 853.297963] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 853.988327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 854.041976] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 856.536556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 856.576193] LNet: 44837:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 856.588535] LNetError: 44837:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 856.597587] LNet: Removed LNI 10.1.2.3@tcp [ 856.621888] LNet: Removed LNI 192.168.206.30@tcp [ 856.651708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 856.743075] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 856.774254] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 856.781689] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 857.291474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 857.325579] LNet: 44864:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 857.337429] LNetError: 44864:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 857.343934] LNetError: 44864:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 857.351677] LNet: Removed LNI 10.1.2.3@tcp [ 857.367494] LNet: Removed LNI 192.168.206.30@tcp [ 857.396178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 857.523980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 857.582479] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 857.589145] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 858.182113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 858.217371] LNet: Added LNI 192.168.206.30@tcp2 [8/256/0/180] [ 859.557808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 859.576396] LNet: 45026:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 859.584134] LNetError: 45026:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 859.593883] LNetError: 45026:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 859.599562] LNet: Removed LNI 10.1.2.3@tcp [ 859.601760] LNet: Removed LNI 192.168.206.30@tcp [ 859.614540] LNet: Removed LNI 192.168.206.30@tcp2 [ 859.665685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 859.726255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 859.744521] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 859.748914] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 859.753838] LNet: Added LNI 192.168.206.30@tcp2 [8/256/0/180] [ 867.073839] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 16:53:48 (1787864028) [ 867.275504] LNet: 45623:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 867.284865] LNetError: 45623:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 867.290508] LNetError: 45623:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 867.296551] LNet: Removed LNI 10.1.2.3@tcp [ 867.298794] LNet: Removed LNI 192.168.206.30@tcp [ 867.309176] LNet: Removed LNI 192.168.206.30@tcp2 [ 868.123082] Key type .llcrypt unregistered [ 868.128540] Key type ._llcrypt unregistered [ 873.919872] Key type ._llcrypt registered [ 873.923905] Key type .llcrypt registered [ 874.249778] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 874.271702] alg: No test for adler32 (adler32-zlib) [ 875.325594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 875.394386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 875.443708] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 876.373818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 876.395815] LNet: 46022:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 876.404456] LNetError: 46022:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 876.420386] LNet: Removed LNI 192.168.206.30@tcp [ 876.463331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 876.520268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 876.540942] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 876.987908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 877.017438] LNet: 46052:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 877.029766] LNetError: 46052:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 877.049574] LNet: Removed LNI 192.168.206.30@tcp [ 877.092431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 877.153492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 877.189773] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 877.728697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 877.772756] LNet: Removed LNI 192.168.206.30@tcp [ 877.794321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 877.851794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 877.900573] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 878.499442] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 878.719453] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 878.727896] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 878.733667] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 881.199705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 881.247675] LNet: 46155:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 881.254995] LNet: 46155:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 881.277490] LNetError: 46155:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 881.283476] LNetError: 46155:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 881.304609] LNet: Removed LNI 192.168.206.30@tcp [ 881.317922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 881.394254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 881.425956] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 888.583344] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 16:54:09 (1787864049) [ 888.791476] LNet: 46747:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 888.797776] LNetError: 46747:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 888.812056] LNet: Removed LNI 192.168.206.30@tcp [ 889.587514] Key type .llcrypt unregistered [ 889.589357] Key type ._llcrypt unregistered [ 896.069723] Key type ._llcrypt registered [ 896.071948] Key type .llcrypt registered [ 896.263980] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 896.275960] alg: No test for adler32 (adler32-zlib) [ 897.281948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 897.408955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 897.456459] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 898.306976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 898.327664] LNet: 47155:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 898.333850] LNetError: 47155:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 898.349638] LNet: Removed LNI 192.168.206.30@tcp [ 898.375433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 898.519876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 898.581622] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 905.496728] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 16:54:26 (1787864066) [ 905.792715] LNet: 47749:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 905.798580] LNetError: 47749:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 905.823605] LNet: Removed LNI 192.168.206.30@tcp [ 906.459805] Key type .llcrypt unregistered [ 906.464906] Key type ._llcrypt unregistered [ 913.542317] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 913.755233] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 913.787048] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 913.798115] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 916.400318] Key type ._llcrypt registered [ 916.402587] Key type .llcrypt registered [ 916.673938] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 916.699399] alg: No test for adler32 (adler32-zlib) [ 917.704638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 917.783776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 917.804949] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 918.679215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 918.721478] LNet: 48205:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 918.731169] LNetError: 48205:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 918.749970] LNet: Removed LNI 192.168.206.30@tcp [ 918.774305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 918.922203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 918.984301] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 926.030233] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 16:54:47 (1787864087) [ 926.330453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 926.353070] LNet: 48803:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 926.359303] LNetError: 48803:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 926.400605] LNet: Removed LNI 192.168.206.30@tcp [ 926.454259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 926.601464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 933.828270] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 16:54:54 (1787864094) [ 934.197420] LNet: 49386:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 934.950856] Key type .llcrypt unregistered [ 934.958254] Key type ._llcrypt unregistered [ 941.715299] Key type ._llcrypt registered [ 941.717456] Key type .llcrypt registered [ 941.962958] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 941.995294] alg: No test for adler32 (adler32-zlib) [ 942.954851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 943.060672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 943.099643] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 943.890814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 943.911630] LNet: 49787:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 943.917908] LNetError: 49787:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 943.941238] LNet: Removed LNI 192.168.206.30@tcp [ 943.977095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 944.025556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 944.046448] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 950.599320] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 16:55:11 (1787864111) [ 950.821594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 950.842919] LNet: 50381:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 950.853285] LNetError: 50381:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 950.870248] LNet: Removed LNI 192.168.206.30@tcp [ 950.896340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 950.960414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 957.164293] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 16:55:18 (1787864118) [ 959.079588] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 960.859038] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 16:55:21 (1787864121) [ 962.174438] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 963.602610] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 16:55:24 (1787864124) [ 965.082350] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 966.619540] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 16:55:27 (1787864127) [ 968.443132] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 970.260276] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 16:55:31 (1787864131) [ 971.935579] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 973.767402] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing ========================================================== 16:55:34 (1787864134) [ 973.900204] LNet: 52425:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 974.638990] Key type .llcrypt unregistered [ 974.643270] Key type ._llcrypt unregistered [ 981.359649] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 981.535435] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 981.546179] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 981.554443] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 984.299516] Key type ._llcrypt registered [ 984.307256] Key type .llcrypt registered [ 984.579458] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 984.613650] alg: No test for adler32 (adler32-zlib) [ 985.580444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 985.683940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 985.732685] LNet: Added LNI 192.168.206.30@tcp [8/512/0/180] [ 985.740220] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 985.743123] LNet: 52757:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 985.851866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 985.876093] LNet: 52776:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 985.883302] LNetError: 52776:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 985.896766] LNet: Removed LNI 10.1.2.3@tcp [ 985.898959] LNetError: 52776:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 985.911859] LNet: Removed LNI 192.168.206.30@tcp [ 985.958047] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 986.078190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 986.112033] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 986.245436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 986.275978] LNet: 52800:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 986.288072] LNetError: 52800:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 986.306134] LNet: Removed LNI 192.168.206.30@tcp [ 986.341645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 986.415240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 986.438856] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 986.445933] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 986.448032] LNet: 52808:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 986.536955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 986.562330] LNet: 52826:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 986.567869] LNetError: 52826:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 986.574149] LNet: Removed LNI 10.1.2.3@tcp [ 986.576440] LNetError: 52826:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 986.587615] LNet: Removed LNI 192.168.206.30@tcp [ 986.654954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 986.754428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 986.778960] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 986.788867] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 986.794289] LNet: 52834:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 987.021772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 987.037988] LNet: 52860:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 987.043744] LNetError: 52860:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 987.049285] LNet: Removed LNI 10.1.2.3@tcp [ 987.053264] LNetError: 52860:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 987.069840] LNet: Removed LNI 192.168.206.30@tcp [ 987.085979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 987.164640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 987.194885] LNet: Added LNI 192.168.206.30@tcp [8/256/0/182] [ 987.210093] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 987.321435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 987.344920] LNet: 52888:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 987.352339] LNetError: 52888:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 987.358549] LNet: Removed LNI 10.1.2.3@tcp [ 987.359832] LNetError: 52888:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 987.379141] LNet: Removed LNI 192.168.206.30@tcp [ 987.410415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 987.490871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 987.515110] LNet: Added LNI 192.168.206.30@tcp [8/256/0/182] [ 987.518682] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 987.521804] LNet: 52896:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 987.605537] LNet: Removed LNI 10.1.2.3@tcp [ 987.616465] LNet: Removed LNI 192.168.206.30@tcp [ 988.499422] Key type .llcrypt unregistered [ 988.501671] Key type ._llcrypt unregistered [ 1001.628606] Key type ._llcrypt registered [ 1001.630592] Key type .llcrypt registered [ 1003.102923] Lustre: DEBUG MARKER: == sanity-lnet test 180: FastReg map error should not leak descriptors ========================================================== 16:56:03 (1787864163) [ 1004.692532] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_180 Need o2ib NETTYPE [ 1006.613951] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 16:56:07 (1787864167) [ 1007.077214] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1007.106889] alg: No test for adler32 (adler32-zlib) [ 1008.049387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1008.110695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1008.151351] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 1008.939639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.206.30 [ 1008.965768] LNet: 54167:0:(config.c:1738:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1008.985079] LNet: Removed LNI 192.168.206.30@tcp [ 1015.717638] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 16:56:16 (1787864176) [ 1016.126790] LNet: 54737:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1016.143453] LNetError: 54737:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1016.171910] LNet: Removed LNI 192.168.206.30@tcp [ 1016.786589] Key type .llcrypt unregistered [ 1016.788272] Key type ._llcrypt unregistered [ 1023.117341] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1023.396546] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1026.116841] Key type ._llcrypt registered [ 1026.120688] Key type .llcrypt registered [ 1026.361292] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1026.380483] alg: No test for adler32 (adler32-zlib) [ 1027.804441] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1034.323992] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 16:56:35 (1787864195) [ 1034.500384] LNet: 55654:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1034.521558] LNetError: 55654:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1034.537886] LNet: Removed LNI 10.1.2.3@tcp [ 1034.990423] Key type .llcrypt unregistered [ 1034.997381] Key type ._llcrypt unregistered [ 1040.832303] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1041.063420] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1041.067555] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1041.074705] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1043.693961] Key type ._llcrypt registered [ 1043.700338] Key type .llcrypt registered [ 1043.956656] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1043.978584] alg: No test for adler32 (adler32-zlib) [ 1044.925980] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1052.104296] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 16:56:52 (1787864212) [ 1054.107885] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 1056.017452] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 16:56:56 (1787864216) [ 1056.367373] LNet: 56843:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1056.373350] LNetError: 56843:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1056.399104] LNet: Removed LNI 10.1.2.3@tcp [ 1057.122511] Key type .llcrypt unregistered [ 1057.126896] Key type ._llcrypt unregistered [ 1064.508608] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1064.753770] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1064.759315] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1064.768279] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1067.500886] Key type ._llcrypt registered [ 1067.506695] Key type .llcrypt registered [ 1067.723654] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1067.742022] alg: No test for adler32 (adler32-zlib) [ 1068.713361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1068.777398] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1078.167847] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 16:57:19 (1787864239) [ 1078.746160] LNet: 58078:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1078.753856] LNetError: 58078:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1078.777136] LNet: Removed LNI 10.1.2.3@tcp [ 1079.662590] Key type .llcrypt unregistered [ 1079.667388] Key type ._llcrypt unregistered [ 1086.680552] Key type ._llcrypt registered [ 1086.684430] Key type .llcrypt registered [ 1086.849218] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1086.870205] alg: No test for adler32 (adler32-zlib) [ 1087.761608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1087.795805] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 1095.482262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.130@tcp [ 1096.262781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.30@tcp -d 192.168.206.130@tcp -m GET -e local_interrupt -r 1 [ 1096.312973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp -d 192.168.206.30@tcp -m GET -e local_interrupt -r 1 [ 1096.384618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.130@tcp [ 1096.413641] LNet: There was an unexpected network error while writing to 192.168.206.130: rc = -22 [ 1096.649322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1096.677544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1096.725574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1096.752767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1096.786596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1096.820960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1096.851391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1096.886430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1096.911615] LNet: 58519:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1096.932466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1096.973599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1097.026874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1097.064165] LNet: 1 local NIs in recovery (showing 1): 192.168.206.30@tcp [ 1097.064876] LNet: There was an unexpected network error while writing to 192.168.206.30: rc = -22 [ 1097.099354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1097.143260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1097.179849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1097.221259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1097.828838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp -d 192.168.206.130@tcp -m GET -e local_dropped -r 1 [ 1097.869741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.30@tcp -d 192.168.206.30@tcp -m GET -e local_dropped -r 1 [ 1097.924737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.130@tcp [ 1098.216347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1098.262493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1098.289775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1098.316615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1098.343676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1098.379559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1098.404531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1098.442205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1098.461227] LNet: 58629:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1098.481492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1098.527512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1098.571319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1098.617179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1098.681803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1098.753332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1098.806772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1099.465402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp -d 192.168.206.130@tcp -m GET -e local_aborted -r 1 [ 1099.508288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp -d 192.168.206.30@tcp -m GET -e local_aborted -r 1 [ 1099.576657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.130@tcp [ 1099.601593] LNet: There was an unexpected network error while writing to 192.168.206.130: rc = -22 [ 1099.610608] LNet: Skipped 1 previous similar message [ 1099.900881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1099.932470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1099.962524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1099.988389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1100.021851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1100.053748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1100.107125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1100.207847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1100.230547] LNet: 58738:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1100.245707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1100.275742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1100.316737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1100.350757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1100.403764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1100.458706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1100.507351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1101.073422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.30@tcp -d 192.168.206.130@tcp -m GET -e local_no_route -r 1 [ 1101.128122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.30@tcp -d 192.168.206.30@tcp -m GET -e local_no_route -r 1 [ 1101.193809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.130@tcp [ 1101.454983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1101.498636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1101.519143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1101.545815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1101.595299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1101.635476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1101.661922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1101.691833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1101.718922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1101.780977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1101.821463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1101.851873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1101.880941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1101.908096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1101.936205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1102.485200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.30@tcp -d 192.168.206.130@tcp -m GET -e local_timeout -r 1 [ 1102.532647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.30@tcp -d 192.168.206.30@tcp -m GET -e local_timeout -r 1 [ 1102.573246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.130@tcp [ 1102.596997] LNet: There was an unexpected network error while writing to 192.168.206.130: rc = -22 [ 1102.606957] LNet: Skipped 1 previous similar message [ 1102.958981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1102.989356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1103.039617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1103.083831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1103.110504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1103.171564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1103.209585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1103.257537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1103.290165] LNet: 58957:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1103.294750] LNet: 58957:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1103.324092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1103.371363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1103.406187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1103.455728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1103.490715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1103.540465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1103.572729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1104.167403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp -d 192.168.206.130@tcp -m GET -e local_error -r 1 [ 1104.209335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.30@tcp -d 192.168.206.30@tcp -m GET -e local_error -r 1 [ 1104.266230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.130@tcp [ 1104.539931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1104.579934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1104.642914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1104.682979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1104.716841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1104.764126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1104.805153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1104.839630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1104.881407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1104.925585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1104.962950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1105.006565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1105.036674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1105.073694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1105.103602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1105.427204] LNet: 59111:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1105.437056] LNetError: 59111:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1105.460956] LNet: Removed LNI 192.168.206.30@tcp [ 1105.879871] Key type .llcrypt unregistered [ 1105.882103] Key type ._llcrypt unregistered [ 1124.757638] Key type ._llcrypt registered [ 1124.759450] Key type .llcrypt registered [ 1125.807494] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 16:58:07 (1787864287) [ 1126.897520] Key type .llcrypt unregistered [ 1126.902439] Key type ._llcrypt unregistered [ 1132.545553] Key type ._llcrypt registered [ 1132.547747] Key type .llcrypt registered [ 1132.856216] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1132.871783] alg: No test for adler32 (adler32-zlib) [ 1133.837514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1133.882661] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 1141.068391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.130@tcp [ 1152.367329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1152.435677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1152.486490] LNet: Added LNI 192.168.206.30@tcp1 [8/256/0/180] [ 1154.060248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp -d 192.168.206.130@tcp -m GET -e local_interrupt -r 1 [ 1154.082748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp -d 192.168.206.130@tcp1 -m GET -e local_interrupt -r 1 [ 1154.117994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.30@tcp -d 192.168.206.30@tcp -m GET -e local_interrupt -r 1 [ 1154.159700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp -d 192.168.206.30@tcp1 -m GET -e local_interrupt -r 1 [ 1154.201781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.30@tcp1 -d 192.168.206.130@tcp -m GET -e local_interrupt -r 1 [ 1154.251541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp1 -d 192.168.206.130@tcp1 -m GET -e local_interrupt -r 1 [ 1154.318954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.30@tcp1 -d 192.168.206.30@tcp -m GET -e local_interrupt -r 1 [ 1154.396641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.30@tcp1 -d 192.168.206.30@tcp1 -m GET -e local_interrupt -r 1 [ 1154.448090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.130@tcp [ 1154.462654] LNet: There was an unexpected network error while writing to 192.168.206.130: rc = -22 [ 1154.470267] LNet: 1 local NIs in recovery (showing 1): 192.168.206.30@tcp [ 1154.722266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1154.763494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1154.805255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1154.855824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1154.906954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1154.951420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1155.001376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1155.040910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1155.076972] LNet: 60586:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1155.106257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1155.169438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1155.220414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1155.294151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1155.343149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1155.400931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1155.444132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1156.167355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.30@tcp -d 192.168.206.130@tcp -m GET -e local_dropped -r 1 [ 1156.225923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp -d 192.168.206.130@tcp1 -m GET -e local_dropped -r 1 [ 1156.292947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.30@tcp -d 192.168.206.30@tcp -m GET -e local_dropped -r 1 [ 1156.340046] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp -d 192.168.206.30@tcp1 -m GET -e local_dropped -r 1 [ 1156.397688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp1 -d 192.168.206.130@tcp -m GET -e local_dropped -r 1 [ 1156.454409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp1 -d 192.168.206.130@tcp1 -m GET -e local_dropped -r 1 [ 1156.497938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp1 -d 192.168.206.30@tcp -m GET -e local_dropped -r 1 [ 1156.579729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.30@tcp1 -d 192.168.206.30@tcp1 -m GET -e local_dropped -r 1 [ 1156.645333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.130@tcp [ 1156.691931] LNet: There was an unexpected network error while writing to 192.168.206.130: rc = -22 [ 1156.705632] LNet: Skipped 2 previous similar messages [ 1156.898646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1156.944328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1156.990454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1157.050584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1157.121252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1157.242514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1157.303070] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1157.383943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1157.414199] LNet: 60707:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1157.465317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1157.528841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1157.564830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1157.607202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1157.635936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1157.697710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1157.773888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1158.462240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp -d 192.168.206.130@tcp -m GET -e local_aborted -r 1 [ 1158.549319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp -d 192.168.206.130@tcp1 -m GET -e local_aborted -r 1 [ 1158.597391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp -d 192.168.206.30@tcp -m GET -e local_aborted -r 1 [ 1158.678770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp -d 192.168.206.30@tcp1 -m GET -e local_aborted -r 1 [ 1158.742944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp1 -d 192.168.206.130@tcp -m GET -e local_aborted -r 1 [ 1158.807362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp1 -d 192.168.206.130@tcp1 -m GET -e local_aborted -r 1 [ 1158.852522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.30@tcp1 -d 192.168.206.30@tcp -m GET -e local_aborted -r 1 [ 1158.913748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp1 -d 192.168.206.30@tcp1 -m GET -e local_aborted -r 1 [ 1158.968077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.130@tcp [ 1159.004319] LNet: There was an unexpected network error while writing to 192.168.206.130: rc = -22 [ 1159.025877] LNet: Skipped 2 previous similar messages [ 1159.296070] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1159.330970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1159.355819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1159.397355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1159.432696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1159.470795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1159.512400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1159.552402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1159.591746] LNet: 60829:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1159.618235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1159.665258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1159.695900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1159.724043] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1159.763126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1159.800313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1159.826792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1160.446723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.30@tcp -d 192.168.206.130@tcp -m GET -e local_no_route -r 1 [ 1160.508534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.30@tcp -d 192.168.206.130@tcp1 -m GET -e local_no_route -r 1 [ 1160.545991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp -d 192.168.206.30@tcp -m GET -e local_no_route -r 1 [ 1160.597306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp -d 192.168.206.30@tcp1 -m GET -e local_no_route -r 1 [ 1160.649348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.30@tcp1 -d 192.168.206.130@tcp -m GET -e local_no_route -r 1 [ 1160.693755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.30@tcp1 -d 192.168.206.130@tcp1 -m GET -e local_no_route -r 1 [ 1160.744082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.30@tcp1 -d 192.168.206.30@tcp -m GET -e local_no_route -r 1 [ 1160.795583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp1 -d 192.168.206.30@tcp1 -m GET -e local_no_route -r 1 [ 1160.875339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.130@tcp [ 1161.248893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1161.289433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1161.344461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1161.406526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1161.481447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1161.547178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1161.586743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1161.630289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1161.654847] LNet: 60950:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1161.684337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1161.713507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1161.738128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1161.765814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1161.789678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1161.813623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1161.843425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1162.363802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp -d 192.168.206.130@tcp -m GET -e local_timeout -r 1 [ 1162.393517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.30@tcp -d 192.168.206.130@tcp1 -m GET -e local_timeout -r 1 [ 1162.430218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp -d 192.168.206.30@tcp -m GET -e local_timeout -r 1 [ 1162.490217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.30@tcp -d 192.168.206.30@tcp1 -m GET -e local_timeout -r 1 [ 1162.545970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.30@tcp1 -d 192.168.206.130@tcp -m GET -e local_timeout -r 1 [ 1162.594566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.30@tcp1 -d 192.168.206.130@tcp1 -m GET -e local_timeout -r 1 [ 1162.628287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp1 -d 192.168.206.30@tcp -m GET -e local_timeout -r 1 [ 1162.665711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.30@tcp1 -d 192.168.206.30@tcp1 -m GET -e local_timeout -r 1 [ 1162.714059] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.130@tcp [ 1162.731654] LNet: There was an unexpected network error while writing to 192.168.206.130: rc = -22 [ 1162.736160] LNet: Skipped 5 previous similar messages [ 1162.917957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1162.950510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1163.004739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1163.064609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1163.113738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1163.147471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1163.182171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1163.215713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1163.258447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1163.310269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1163.338985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1163.368438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1163.405829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1163.448163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1163.486336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1164.073564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.30@tcp -d 192.168.206.130@tcp -m GET -e local_error -r 1 [ 1164.116429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.30@tcp -d 192.168.206.130@tcp1 -m GET -e local_error -r 1 [ 1164.175250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp -d 192.168.206.30@tcp -m GET -e local_error -r 1 [ 1164.227314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp -d 192.168.206.30@tcp1 -m GET -e local_error -r 1 [ 1164.292111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp1 -d 192.168.206.130@tcp -m GET -e local_error -r 1 [ 1164.355842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.30@tcp1 -d 192.168.206.130@tcp1 -m GET -e local_error -r 1 [ 1164.446815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp1 -d 192.168.206.30@tcp -m GET -e local_error -r 1 [ 1164.511545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.30@tcp1 -d 192.168.206.30@tcp1 -m GET -e local_error -r 1 [ 1164.593740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.130@tcp [ 1164.952372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1165.001549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1165.048572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1165.096928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1165.129949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1165.180359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1165.235463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1165.295164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1165.348654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1165.401290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1165.435302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1165.483716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1165.539243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1165.582928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1165.618607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1171.676685] LNet: 61273:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1171.683057] LNetError: 61273:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1171.694559] LNet: Removed LNI 192.168.206.30@tcp [ 1171.725714] LNet: Removed LNI 192.168.206.30@tcp1 [ 1172.153671] Key type .llcrypt unregistered [ 1172.161335] Key type ._llcrypt unregistered [ 1188.483751] Key type ._llcrypt registered [ 1188.486343] Key type .llcrypt registered [ 1189.630136] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 16:59:10 (1787864350) [ 1190.587879] Key type .llcrypt unregistered [ 1190.591347] Key type ._llcrypt unregistered [ 1196.285463] Key type ._llcrypt registered [ 1196.287077] Key type .llcrypt registered [ 1196.450914] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1196.471253] alg: No test for adler32 (adler32-zlib) [ 1197.410254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1197.448267] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 1204.884459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.130@tcp [ 1205.670074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.30@tcp -d 192.168.206.130@tcp -m GET -e remote_dropped -r 1 [ 1205.739257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp -d 192.168.206.30@tcp -m GET -e remote_dropped -r 1 [ 1205.784513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.130@tcp [ 1205.808404] LNet: There was an unexpected network error while writing to 192.168.206.130: rc = -22 [ 1206.086560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1206.132874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1206.167858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1206.199236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1206.232488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1206.268361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1206.306653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1206.345929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1206.369867] LNet: 62526:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1206.391847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1206.433994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1206.467606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1206.531067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1206.570486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1206.613707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1206.653693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1207.329429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.30@tcp -d 192.168.206.130@tcp -m GET -e remote_error -r 1 [ 1207.369382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.30@tcp -d 192.168.206.30@tcp -m GET -e remote_error -r 1 [ 1207.396840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.130@tcp [ 1207.419483] LNet: There was an unexpected network error while writing to 192.168.206.130: rc = -22 [ 1207.595525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1207.646980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1207.684065] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1207.716772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1207.755865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1207.816613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1207.866924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1207.903025] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1207.936897] LNet: 62635:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1207.974923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1208.013677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1208.066184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1208.105497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1208.136980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1208.164188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1208.201414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1208.792968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp -d 192.168.206.130@tcp -m GET -e remote_timeout -r 1 [ 1208.870428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp -d 192.168.206.30@tcp -m GET -e remote_timeout -r 1 [ 1208.968269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.130@tcp [ 1209.051463] LNet: There was an unexpected network error while writing to 192.168.206.130: rc = -22 [ 1209.404524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1209.462502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1209.527624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1209.563949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1209.626789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1209.701837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1209.759818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1209.804650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1209.835510] LNet: 62744:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1209.872234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1209.921658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1209.969977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1210.056752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1210.132826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1210.204875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1210.261614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1210.686969] LNet: 62789:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1210.719155] LNetError: 62789:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1210.752059] LNet: Removed LNI 192.168.206.30@tcp [ 1211.326644] Key type .llcrypt unregistered [ 1211.330793] Key type ._llcrypt unregistered [ 1230.694562] Key type ._llcrypt registered [ 1230.697958] Key type .llcrypt registered [ 1231.929097] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 16:59:52 (1787864392) [ 1232.897428] Key type .llcrypt unregistered [ 1232.899434] Key type ._llcrypt unregistered [ 1239.245650] Key type ._llcrypt registered [ 1239.250906] Key type .llcrypt registered [ 1239.459394] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1239.476802] alg: No test for adler32 (adler32-zlib) [ 1240.456092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1240.492258] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 1247.750845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.130@tcp [ 1255.951285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1256.020705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1256.068980] LNet: Added LNI 192.168.206.30@tcp1 [8/256/0/180] [ 1257.447975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.30@tcp -d 192.168.206.130@tcp -m GET -e remote_dropped -r 1 [ 1257.482389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.30@tcp -d 192.168.206.130@tcp1 -m GET -e remote_dropped -r 1 [ 1257.516516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp -d 192.168.206.30@tcp -m GET -e remote_dropped -r 1 [ 1257.548603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp -d 192.168.206.30@tcp1 -m GET -e remote_dropped -r 1 [ 1257.596784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp1 -d 192.168.206.130@tcp -m GET -e remote_dropped -r 1 [ 1257.644681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.30@tcp1 -d 192.168.206.130@tcp1 -m GET -e remote_dropped -r 1 [ 1257.685919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.30@tcp1 -d 192.168.206.30@tcp -m GET -e remote_dropped -r 1 [ 1257.718732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp1 -d 192.168.206.30@tcp1 -m GET -e remote_dropped -r 1 [ 1257.777544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.130@tcp [ 1257.795636] LNet: There was an unexpected network error while writing to 192.168.206.130: rc = -22 [ 1257.802188] LNet: 1 peer NIs in recovery (showing 1): 192.168.206.130@tcp [ 1258.056518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1258.087940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1258.114673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1258.141845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1258.171675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1258.210551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1258.239851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1258.270980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1258.287402] LNet: 64261:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1258.308674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1258.339943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1258.378701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1258.417835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1258.460290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1258.492258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1258.535479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1258.588663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1259.245983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp -d 192.168.206.130@tcp -m GET -e remote_error -r 1 [ 1259.298676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.30@tcp -d 192.168.206.130@tcp1 -m GET -e remote_error -r 1 [ 1259.336579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp -d 192.168.206.30@tcp -m GET -e remote_error -r 1 [ 1259.368842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.30@tcp -d 192.168.206.30@tcp1 -m GET -e remote_error -r 1 [ 1259.398746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.30@tcp1 -d 192.168.206.130@tcp -m GET -e remote_error -r 1 [ 1259.424645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp1 -d 192.168.206.130@tcp1 -m GET -e remote_error -r 1 [ 1259.455480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.30@tcp1 -d 192.168.206.30@tcp -m GET -e remote_error -r 1 [ 1259.510933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp1 -d 192.168.206.30@tcp1 -m GET -e remote_error -r 1 [ 1259.567655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.130@tcp [ 1259.610046] LNet: There was an unexpected network error while writing to 192.168.206.130: rc = -22 [ 1259.620825] LNet: Skipped 2 previous similar messages [ 1259.906368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1259.940495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1259.975535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1260.020102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1260.046099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1260.095902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1260.140948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1260.172413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1260.186759] LNet: 64384:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1260.215284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1260.259397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1260.288735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1260.333264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1260.365446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1260.388150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1260.412290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1260.442843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1260.877921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp -d 192.168.206.130@tcp -m GET -e remote_timeout -r 1 [ 1260.921505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp -d 192.168.206.130@tcp1 -m GET -e remote_timeout -r 1 [ 1260.986455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.30@tcp -d 192.168.206.30@tcp -m GET -e remote_timeout -r 1 [ 1261.033938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.30@tcp -d 192.168.206.30@tcp1 -m GET -e remote_timeout -r 1 [ 1261.072668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.30@tcp1 -d 192.168.206.130@tcp -m GET -e remote_timeout -r 1 [ 1261.111946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.30@tcp1 -d 192.168.206.130@tcp1 -m GET -e remote_timeout -r 1 [ 1261.148773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp1 -d 192.168.206.30@tcp -m GET -e remote_timeout -r 1 [ 1261.190418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp1 -d 192.168.206.30@tcp1 -m GET -e remote_timeout -r 1 [ 1261.236545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.130@tcp [ 1261.257642] LNet: There was an unexpected network error while writing to 192.168.206.130: rc = -22 [ 1261.477043] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1261.506145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1261.543976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1261.573543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1261.603360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1261.647868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1261.684071] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1261.731825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1261.752973] LNet: 64508:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1261.785950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1261.831691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1261.881461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1261.916183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1261.941776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1262.017594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1262.046218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1262.093127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1266.891961] LNet: 64590:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1266.898948] LNetError: 64590:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1266.906821] LNet: Removed LNI 192.168.206.30@tcp [ 1266.914737] LNet: Removed LNI 192.168.206.30@tcp1 [ 1267.322610] Key type .llcrypt unregistered [ 1267.324669] Key type ._llcrypt unregistered [ 1282.656333] Key type ._llcrypt registered [ 1282.658146] Key type .llcrypt registered [ 1283.805749] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 17:00:44 (1787864444) [ 1285.727447] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 1287.338188] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 17:00:48 (1787864448) [ 1288.285418] Key type .llcrypt unregistered [ 1288.291266] Key type ._llcrypt unregistered [ 1294.647538] Key type ._llcrypt registered [ 1294.650292] Key type .llcrypt registered [ 1294.851635] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1294.865873] alg: No test for adler32 (adler32-zlib) [ 1295.793465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1295.821288] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 1303.241478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.130@tcp [ 1304.172520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp -d 192.168.206.130@tcp -m GET -e network_timeout -r 1 [ 1304.232602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.30@tcp -d 192.168.206.30@tcp -m GET -e network_timeout -r 1 [ 1304.291472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.130@tcp [ 1304.312992] LNet: There was an unexpected network error while writing to 192.168.206.130: rc = -22 [ 1304.585935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1304.641136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1304.698610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1304.747835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1304.820242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1304.877402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1304.927244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1304.965317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1304.999636] LNet: 66151:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1305.023965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1305.057583] LNet: 1 local NIs in recovery (showing 1): 192.168.206.30@tcp [ 1305.065104] LNet: There was an unexpected network error while writing to 192.168.206.30: rc = -22 [ 1305.109278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1305.155869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1305.206069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1305.241701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1305.268964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1305.557948] LNet: 66194:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1305.567740] LNetError: 66194:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1305.588282] LNet: Removed LNI 192.168.206.30@tcp [ 1305.992665] Key type .llcrypt unregistered [ 1305.999761] Key type ._llcrypt unregistered [ 1325.354900] Key type ._llcrypt registered [ 1325.357161] Key type .llcrypt registered [ 1325.462460] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1325.478205] alg: No test for adler32 (adler32-zlib) [ 1326.403688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1326.435512] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 1335.051295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.130@tcp [ 1344.340938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1344.373800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1344.405791] LNet: Added LNI 192.168.206.30@tcp1 [8/256/0/180] [ 1345.904280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp -d 192.168.206.130@tcp -m GET -e network_timeout -r 1 [ 1345.960203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.30@tcp -d 192.168.206.130@tcp1 -m GET -e network_timeout -r 1 [ 1345.998271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.30@tcp -d 192.168.206.30@tcp -m GET -e network_timeout -r 1 [ 1346.056812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.30@tcp -d 192.168.206.30@tcp1 -m GET -e network_timeout -r 1 [ 1346.101830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp1 -d 192.168.206.130@tcp -m GET -e network_timeout -r 1 [ 1346.148723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp1 -d 192.168.206.130@tcp1 -m GET -e network_timeout -r 1 [ 1346.227955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp1 -d 192.168.206.30@tcp -m GET -e network_timeout -r 1 [ 1346.282157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp1 -d 192.168.206.30@tcp1 -m GET -e network_timeout -r 1 [ 1346.321751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.130@tcp [ 1346.345694] LNet: There was an unexpected network error while writing to 192.168.206.130: rc = -22 [ 1346.557162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1346.596087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1346.632742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1346.671979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1346.699214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1346.733282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1346.769345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1346.813586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1346.830665] LNet: 67079:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1346.850910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1346.890261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1346.913429] LNet: 1 local NIs in recovery (showing 1): 192.168.206.30@tcp [ 1346.918500] LNet: 1 peer NIs in recovery (showing 1): 192.168.206.130@tcp [ 1346.934924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1346.964935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1346.992830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1347.051644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1351.517175] LNet: There was an unexpected network error while writing to 192.168.206.130: rc = -22 [ 1351.842880] LNet: 67157:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1351.851760] LNetError: 67157:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1351.865476] LNet: Removed LNI 192.168.206.30@tcp [ 1351.880406] LNet: Removed LNI 192.168.206.30@tcp1 [ 1352.251608] Key type .llcrypt unregistered [ 1352.253426] Key type ._llcrypt unregistered [ 1368.513269] Key type ._llcrypt registered [ 1368.516751] Key type .llcrypt registered [ 1369.717511] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 17:02:10 (1787864530) [ 1370.190941] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1370.215969] alg: No test for adler32 (adler32-zlib) [ 1371.115562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1371.149356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1371.166344] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 1371.890895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1371.911893] LNet: Added LNI 192.168.206.30@tcp1 [8/256/0/180] [ 1372.637651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.30@tcp [ 1372.694368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1372.725696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1372.760541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1372.785411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1372.808469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1372.862687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.206.30@tcp [ 1372.880427] LNetError: 68259:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.206.30@tcp to recovery [ 1373.151429] LNet: 1 local NIs in recovery (showing 1): 192.168.206.30@tcp [ 1373.151510] LNet: There was an unexpected network error while writing to 192.168.206.30: rc = -22 [ 1375.208540] LNet: There was an unexpected network error while writing to 192.168.206.30: rc = -22 [ 1376.418725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1379.295419] LNet: There was an unexpected network error while writing to 192.168.206.30: rc = -22 [ 1379.855365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1379.884394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1379.922819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1379.936526] LNet: 68366:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1379.940882] LNetError: 68366:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1380.959276] LNet: Removed LNI 192.168.206.30@tcp [ 1380.962830] LNetError: 68366:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1380.997344] LNet: Removed LNI 192.168.206.30@tcp1 [ 1381.041479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1381.156478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1381.182445] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 1381.660540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1381.676407] LNet: Added LNI 192.168.206.30@tcp1 [8/256/0/180] [ 1382.400350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.30@tcp [ 1382.444851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1382.537131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1382.592177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1382.654588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1382.699268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1382.725441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1382.756440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.206.30@tcp [ 1382.780167] LNetError: 68639:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.206.30@tcp to recovery [ 1383.136546] LNet: There was an unexpected network error while writing to 192.168.206.30: rc = -22 [ 1386.298967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1389.279596] LNet: There was an unexpected network error while writing to 192.168.206.30: rc = -22 [ 1389.287743] LNet: Skipped 1 previous similar message [ 1394.316123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1394.370994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1394.429870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1401.144778] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 17:02:42 (1787864562) [ 1401.452952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1401.481588] LNet: 69356:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1401.496581] LNetError: 69356:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1401.506597] LNet: Removed LNI 192.168.206.30@tcp [ 1401.524948] LNet: Removed LNI 192.168.206.30@tcp1 [ 1401.551625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1401.661572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1401.711618] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 1402.520649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1402.564044] LNet: Added LNI 192.168.206.30@tcp1 [8/256/0/180] [ 1403.379540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.30@tcp [ 1403.456341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1403.500936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1403.539285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1403.581929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1403.611279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1403.642381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.206.30@tcp --health 0 [ 1403.679123] LNet: 1 peer NIs in recovery (showing 1): 192.168.206.30@tcp [ 1403.768836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1404.710607] LNet: There was an unexpected network error while writing to 192.168.206.30: rc = -22 [ 1404.724152] LNet: Skipped 1 previous similar message [ 1419.104777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1419.278261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1419.315586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1419.356711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.206.30@tcp --health 500 [ 1419.488820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1426.205580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1426.286178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1426.306407] LNet: 69847:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1426.315037] LNetError: 69847:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1426.323634] LNetError: 69847:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1426.332164] LNet: Removed LNI 192.168.206.30@tcp [ 1426.353655] LNet: Removed LNI 192.168.206.30@tcp1 [ 1426.382508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1426.427492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1426.457590] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 1427.094381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1427.111813] LNet: Added LNI 192.168.206.30@tcp1 [8/256/0/180] [ 1427.910609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.30@tcp [ 1427.945949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1428.002577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1428.038174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1428.079114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1428.133879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1428.171678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1428.212857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.206.30@tcp --health 0 [ 1429.471637] LNet: There was an unexpected network error while writing to 192.168.206.30: rc = -22 [ 1429.480037] LNet: Skipped 3 previous similar messages [ 1430.643634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1440.863842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1440.908554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1440.955490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1447.226819] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 17:03:28 (1787864608) [ 1447.816518] LNet: 70907:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1447.822259] LNetError: 70907:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1447.829113] LNetError: 70907:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1447.835210] LNet: Removed LNI 192.168.206.30@tcp [ 1447.858545] LNet: Removed LNI 192.168.206.30@tcp1 [ 1448.652691] Key type .llcrypt unregistered [ 1448.655627] Key type ._llcrypt unregistered [ 1455.215501] Key type ._llcrypt registered [ 1455.220436] Key type .llcrypt registered [ 1455.504840] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1455.533916] alg: No test for adler32 (adler32-zlib) [ 1456.515344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1456.540799] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 1466.272483] Lustre: DEBUG MARKER: Initial discovery [ 1466.323170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.206.130@tcp [ 1469.172869] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1469.245511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1469.292366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.206.130@tcp [ 1469.333370] LNet: There was an unexpected network error while writing to 192.168.206.130: rc = -22 [ 1469.375280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1469.418583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.206.130@tcp -r 1 -m GET -l 3 [ 1469.445972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.206.130@tcp -r 1 -m GET -e local_error [ 1469.480716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.206.130@tcp -r 1 -m PUT -l 6 [ 1469.856732] LNet: 1 local NIs in recovery (showing 1): 192.168.206.30@tcp [ 1470.812321] Lustre: DEBUG MARKER: Force oleg630-client.virtnet to discover 192.168.206.130@tcp (in background) [ 1470.883498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.206.130@tcp [ 1472.205797] Lustre: DEBUG MARKER: Force oleg630-server to discover 192.168.206.30@tcp [ 1473.951352] LNet: There was an unexpected network error while writing to 192.168.206.130: rc = -22 [ 1474.610428] Lustre: DEBUG MARKER: Wait for 71665 [ 1476.164775] Lustre: DEBUG MARKER: Finished wait on 71665 [ 1476.229225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay del -a [ 1476.256430] LNet: There was an unexpected network error while writing to 192.168.206.130: rc = -22 [ 1476.274621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1476.679123] LNet: 72127:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1476.685240] LNetError: 72127:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1476.704943] LNet: Removed LNI 192.168.206.30@tcp [ 1477.167571] Key type .llcrypt unregistered [ 1477.170804] Key type ._llcrypt unregistered [ 1492.282710] Key type ._llcrypt registered [ 1492.285575] Key type .llcrypt registered [ 1493.377596] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 17:04:14 (1787864654) [ 1493.999345] Key type .llcrypt unregistered [ 1494.003685] Key type ._llcrypt unregistered [ 1499.380796] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1499.575696] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1499.584493] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1499.600863] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1502.225497] Key type ._llcrypt registered [ 1502.227837] Key type .llcrypt registered [ 1502.523115] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1502.541689] alg: No test for adler32 (adler32-zlib) [ 1503.465823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1503.539431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1503.563826] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 1504.274466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1504.300687] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1512.075407] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 17:04:33 (1787864673) [ 1512.218046] LNet: 74201:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1512.224913] LNetError: 74201:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1512.236853] LNet: Removed LNI 10.1.2.3@tcp [ 1512.257975] LNet: Removed LNI 192.168.206.30@tcp [ 1512.914926] Key type .llcrypt unregistered [ 1512.923914] Key type ._llcrypt unregistered [ 1519.251527] Key type ._llcrypt registered [ 1519.253559] Key type .llcrypt registered [ 1519.504991] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1519.529189] alg: No test for adler32 (adler32-zlib) [ 1520.464509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1520.530399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1520.569590] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 1521.374561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1521.395519] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1522.224314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.206.30@tcp [ 1523.295520] LNetError: Unexpected error -22 connecting to 192.168.206.30@tcp at host 192.168.206.30:988 [ 1529.224783] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 17:04:49 (1787864689) [ 1529.359882] LNet: 75338:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1529.373672] LNetError: 75338:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1529.386848] LNet: Removed LNI 10.1.2.3@tcp [ 1530.411172] LNet: Removed LNI 192.168.206.30@tcp [ 1531.298459] Key type .llcrypt unregistered [ 1531.304217] Key type ._llcrypt unregistered [ 1538.872224] Key type ._llcrypt registered [ 1538.874431] Key type .llcrypt registered [ 1539.039902] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1539.047902] alg: No test for adler32 (adler32-zlib) [ 1540.009314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1540.075399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1540.095322] LNet: Added LNI 192.168.206.30@tcp1 [8/256/0/180] [ 1541.003605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1541.022347] LNet: Added LNI 192.168.206.30@tcp2 [8/256/0/180] [ 1542.230597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.206.30@tcp1 --nid 192.168.206.30@tcp2 [ 1546.738307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.30@tcp1 192.168.206.30@tcp2 [ 1546.763072] LNetError: 76184:0:(lib-move.c:2101:lnet_handle_find_routed_path()) Unable to send message from 192.168.206.30@tcp1 to 192.168.206.30@tcp2 - Route table may be misconfigured [ 1546.791426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.30@tcp2 192.168.206.30@tcp1 [ 1552.591868] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 17:05:13 (1787864713) [ 1552.690759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1552.715598] LNet: 76751:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1552.724335] LNetError: 76751:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1552.734284] LNet: Removed LNI 192.168.206.30@tcp1 [ 1553.786866] LNet: Removed LNI 192.168.206.30@tcp2 [ 1553.868893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1553.985965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1554.029186] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 1554.722377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1554.744553] LNet: Added LNI 192.168.206.30@tcp1 [8/256/0/180] [ 1555.553736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.30@tcp [ 1555.614552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.30@tcp 192.168.206.30@tcp [ 1555.665688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.30@tcp1 192.168.206.30@tcp1 [ 1555.725074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.206.30@tcp -d 192.168.206.30@tcp -e network_timeout [ 1555.757797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.206.30@tcp -d 192.168.206.30@tcp1 -e network_timeout [ 1555.799567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.206.30@tcp1 -d 192.168.206.30@tcp -e network_timeout [ 1555.829243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.206.30@tcp1 -d 192.168.206.30@tcp1 -e network_timeout [ 1555.859929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.30@tcp [ 1555.879930] LNet: There was an unexpected network error while writing to 192.168.206.30: rc = -22 [ 1555.897304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1555.972230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1555.999186] LNet: 1 local NIs in recovery (showing 1): 192.168.206.30@tcp [ 1556.057163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1560.994311] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 17:05:22 (1787864722) [ 1561.166113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1561.187348] LNet: 77618:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1561.195226] LNetError: 77618:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1561.199750] LNetError: 77618:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1561.213460] LNet: Removed LNI 192.168.206.30@tcp [ 1562.280338] LNet: Removed LNI 192.168.206.30@tcp1 [ 1562.302891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1562.372752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1562.673633] LNet: 77660:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1563.001091] Key type .llcrypt unregistered [ 1563.004774] Key type ._llcrypt unregistered [ 1572.457260] Key type ._llcrypt registered [ 1572.460741] Key type .llcrypt registered [ 1573.610370] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 17:05:34 (1787864734) [ 1573.910927] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1573.922678] alg: No test for adler32 (adler32-zlib) [ 1574.809781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1575.959244] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1577.449973] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 17:05:38 (1787864738) [ 1577.739210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1577.755748] LNet: 78758:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1577.776896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1577.836298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1577.863315] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 1578.412528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1578.428355] LNet: Added LNI 192.168.206.30@tcp1 [8/256/0/180] [ 1579.121560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.30@tcp [ 1579.171711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.30@tcp1 [ 1579.210108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.30@tcp1 [ 1584.295597] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 17:05:45 (1787864745) [ 1585.782421] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg630-server [ 1587.226949] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 17:05:48 (1787864748) [ 1589.108908] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg630-server [ 1590.605880] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 17:05:51 (1787864751) [ 1592.157912] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg630-server [ 1593.528417] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 17:05:54 (1787864754) [ 1594.988572] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg630-server [ 1596.282390] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 17:05:57 (1787864757) [ 1597.735560] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg630-server [ 1599.447416] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 17:06:00 (1787864760) [ 1601.221055] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg630-server [ 1602.624734] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 17:06:03 (1787864763) [ 1604.286798] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg630-server [ 1605.861716] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 17:06:07 (1787864767) [ 1607.303693] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg630-server [ 1608.832468] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 17:06:10 (1787864770) [ 1610.122114] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg630-server [ 1611.466175] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 17:06:12 (1787864772) [ 1611.571623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1611.604864] LNet: 82631:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1611.610968] LNetError: 82631:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1611.617453] LNet: Removed LNI 192.168.206.30@tcp [ 1612.646824] LNet: Removed LNI 192.168.206.30@tcp1 [ 1612.681582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1612.762958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1612.795859] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 1613.252379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1613.317983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.30@tcp [ 1614.533690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1614.551382] LNet: 82793:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1614.559551] LNetError: 82793:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1614.563781] LNetError: 82793:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1614.574801] LNet: Removed LNI 192.168.206.30@tcp [ 1614.603513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1614.646988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1614.664386] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 1615.399305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.30@tcp [ 1615.581532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1615.600588] LNetError: 82938:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1615.619744] LNet: Removed LNI 192.168.206.30@tcp [ 1615.655713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1615.710781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1615.746531] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 1616.443054] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1616.521346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.30@tcp [ 1617.823739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1617.858508] LNet: 83094:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1617.863230] LNet: 83094:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1617.868724] LNetError: 83094:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1617.883528] LNet: Removed LNI 192.168.206.30@tcp [ 1617.900821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1617.966509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1617.993832] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 1618.617461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.30@tcp [ 1618.809321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1619.889956] LNet: Removed LNI 192.168.206.30@tcp [ 1619.922647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1619.991744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1620.028042] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 1620.647430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1620.726538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.30@tcp [ 1620.903397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1621.992236] LNet: Removed LNI 192.168.206.30@tcp [ 1622.018532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1622.084576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1622.109780] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 1622.828449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.30@tcp [ 1623.040405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1623.064250] LNet: 83533:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1623.070798] LNet: 83533:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 1623.082230] LNetError: 83533:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1623.090917] LNetError: 83533:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1623.112439] LNet: Removed LNI 192.168.206.30@tcp [ 1623.145339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1623.214582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1623.250975] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 1623.846087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1623.928409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.30@tcp [ 1624.108519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1625.189337] LNet: Removed LNI 192.168.206.30@tcp [ 1625.221066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1625.293984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1625.337425] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 1626.004376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.30@tcp [ 1627.406961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1628.457709] LNet: Removed LNI 192.168.206.30@tcp [ 1628.472880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1628.530520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1628.565481] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 1629.078453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1629.155796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.30@tcp [ 1629.392428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1630.437523] LNet: Removed LNI 192.168.206.30@tcp [ 1630.461878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1630.519732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1630.542081] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 1631.165435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.30@tcp [ 1632.477428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1632.491482] LNet: 84136:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1632.495162] LNet: 84136:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 1632.500557] LNetError: 84136:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1632.505304] LNetError: 84136:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 1633.573896] LNet: Removed LNI 192.168.206.30@tcp [ 1633.595219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1633.650682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1633.669837] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 1634.183412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1634.247954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.30@tcp [ 1635.644912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1636.721926] LNet: Removed LNI 192.168.206.30@tcp [ 1636.760046] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1636.817726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1636.864438] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 1637.767117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.30@tcp [ 1638.076022] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1639.148788] LNet: Removed LNI 192.168.206.30@tcp [ 1639.192640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1639.242815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1639.275466] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 1639.846402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1639.894336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1639.928911] LNet: Removed LNI 192.168.206.30@tcp [ 1639.945875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1639.996729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1640.016902] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 1640.768267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1640.844500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1640.879635] LNet: Removed LNI 192.168.206.30@tcp [ 1640.901624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1640.941368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 128 [ 1641.019576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1641.055253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1641.093621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 1000000000000000000000000 [ 1641.168139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1641.213268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1641.270954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer foo [ 1641.353570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1641.387588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1641.439908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 8 [ 1641.459056] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 1642.162904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1642.196958] LNet: Removed LNI 192.168.206.30@tcp [ 1642.218911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1642.272689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1648.373601] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 17:06:49 (1787864809) [ 1648.490964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1648.510510] LNet: 85455:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1648.517873] LNet: 85455:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 8 previous similar messages [ 1648.548040] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1648.603904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1648.636471] LNet: Added LNI 192.168.206.30@tcp231 [8/256/0/180] [ 1649.720587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1649.748452] LNetError: 85596:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1649.758599] LNetError: 85596:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 1649.785335] LNet: Removed LNI 192.168.206.30@tcp231 [ 1649.834492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1649.893552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1649.911427] LNet: Added LNI 192.168.206.30@tcp231 [8/256/0/180] [ 1649.916449] LNet: 85604:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1650.411257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1650.454141] LNet: Removed LNI 192.168.206.30@tcp231 [ 1650.471391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1650.506737] LNet: Added LNI 192.168.206.30@tcp231 [8/256/0/180] [ 1651.626609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1651.658317] LNet: Removed LNI 192.168.206.30@tcp231 [ 1651.677223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1651.756242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1651.792568] LNet: Added LNI 192.168.206.30@tcp231 [8/256/0/0] [ 1651.799792] LNet: 85757:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1652.336577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1652.364574] LNet: Removed LNI 192.168.206.30@tcp231 [ 1652.376399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1652.396610] LNet: Added LNI 192.168.206.30@tcp231 [8/256/0/0] [ 1653.363194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1653.389971] LNet: Removed LNI 192.168.206.30@tcp231 [ 1653.410378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1653.461616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1653.483728] LNet: Added LNI 192.168.206.30@tcp231 [8/256/0/60] [ 1653.492364] LNet: 85908:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1653.905145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1653.935337] LNet: Removed LNI 192.168.206.30@tcp231 [ 1653.951804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1653.984120] LNet: Added LNI 192.168.206.30@tcp231 [8/256/0/60] [ 1655.035412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1655.055782] LNet: Removed LNI 192.168.206.30@tcp231 [ 1655.072564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1655.133492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1655.150147] LNet: Added LNI 192.168.206.30@tcp231 [8/256/0/180] [ 1655.633859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1655.684284] LNet: Removed LNI 192.168.206.30@tcp231 [ 1655.702520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1655.743564] LNet: Added LNI 192.168.206.30@tcp231 [8/256/0/180] [ 1661.643508] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 17:07:02 (1787864822) [ 1661.756450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1661.807642] LNet: Removed LNI 192.168.206.30@tcp231 [ 1661.843763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1661.947510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1661.993424] LNet: Added LNI 192.168.206.30@tcp232 [8/256/0/180] [ 1662.855719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1662.889936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1662.915507] LNet: Removed LNI 192.168.206.30@tcp232 [ 1662.943308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1662.979720] LNet: Added LNI 192.168.206.30@tcp232 [8/256/0/180] [ 1663.578684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1663.630433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1663.682990] LNet: Removed LNI 192.168.206.30@tcp232 [ 1663.710192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1663.734221] LNet: Added LNI 192.168.206.30@tcp232 [8/256/0/180] [ 1664.368257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1664.398913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1664.423751] LNet: Removed LNI 192.168.206.30@tcp232 [ 1664.433952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1664.466105] LNet: Added LNI 192.168.206.30@tcp232 [8/256/0/180] [ 1665.094565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1665.134501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1665.162643] LNet: Removed LNI 192.168.206.30@tcp232 [ 1665.190139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1670.882312] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 17:07:11 (1787864831) [ 1671.856094] Key type .llcrypt unregistered [ 1671.857668] Key type ._llcrypt unregistered [ 1677.386111] Key type ._llcrypt registered [ 1677.388386] Key type .llcrypt registered [ 1677.603232] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1677.616308] alg: No test for adler32 (adler32-zlib) [ 1678.553361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1678.589393] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 1685.323465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.130@tcp [ 1693.072638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1693.131726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1693.146818] LNet: Added LNI 192.168.206.30@tcp1 [8/256/0/180] [ 1694.477585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp -d 192.168.206.130@tcp -m GET -e local_interrupt -r 2 [ 1694.518392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp -d 192.168.206.130@tcp1 -m GET -e local_interrupt -r 2 [ 1694.567436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp -d 192.168.206.30@tcp -m GET -e local_interrupt -r 2 [ 1694.610878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp -d 192.168.206.30@tcp1 -m GET -e local_interrupt -r 2 [ 1694.648258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp1 -d 192.168.206.130@tcp -m GET -e local_interrupt -r 2 [ 1694.676162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.30@tcp1 -d 192.168.206.130@tcp1 -m GET -e local_interrupt -r 2 [ 1694.722099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.30@tcp1 -d 192.168.206.30@tcp -m GET -e local_interrupt -r 2 [ 1694.759784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.30@tcp1 -d 192.168.206.30@tcp1 -m GET -e local_interrupt -r 2 [ 1694.795885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.30@tcp 192.168.206.130@tcp [ 1694.817744] LNet: There was an unexpected network error while writing to 192.168.206.130: rc = -22 [ 1694.822544] LNet: 1 local NIs in recovery (showing 1): 192.168.206.30@tcp [ 1695.045477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1695.078400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1695.118757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1695.158133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1695.193615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1695.241526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1695.278919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1695.311745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1695.330032] LNet: 88563:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1695.344868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1695.377409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1695.409764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1695.440604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1695.487917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1695.535697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1696.012550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.30@tcp -d 192.168.206.130@tcp -m GET -e local_dropped -r 2 [ 1696.062222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.30@tcp -d 192.168.206.130@tcp1 -m GET -e local_dropped -r 2 [ 1696.098961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.30@tcp -d 192.168.206.30@tcp -m GET -e local_dropped -r 2 [ 1696.135680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.30@tcp -d 192.168.206.30@tcp1 -m GET -e local_dropped -r 2 [ 1696.172222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.30@tcp1 -d 192.168.206.130@tcp -m GET -e local_dropped -r 2 [ 1696.238755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp1 -d 192.168.206.130@tcp1 -m GET -e local_dropped -r 2 [ 1696.279396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.30@tcp1 -d 192.168.206.30@tcp -m GET -e local_dropped -r 2 [ 1696.326656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp1 -d 192.168.206.30@tcp1 -m GET -e local_dropped -r 2 [ 1696.389265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.30@tcp 192.168.206.130@tcp [ 1696.435934] LNet: There was an unexpected network error while writing to 192.168.206.130: rc = -22 [ 1696.685445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1696.714465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1696.739505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1696.766904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1696.817187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1696.854804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1696.903813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1696.963922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1696.986943] LNet: 88684:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1697.020335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1697.069815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1697.091760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1697.124790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1697.161368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1697.204856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1697.748651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.30@tcp -d 192.168.206.130@tcp -m GET -e local_aborted -r 2 [ 1697.789723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.30@tcp -d 192.168.206.130@tcp1 -m GET -e local_aborted -r 2 [ 1697.819521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp -d 192.168.206.30@tcp -m GET -e local_aborted -r 2 [ 1697.859266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp -d 192.168.206.30@tcp1 -m GET -e local_aborted -r 2 [ 1697.917256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.30@tcp1 -d 192.168.206.130@tcp -m GET -e local_aborted -r 2 [ 1697.975581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp1 -d 192.168.206.130@tcp1 -m GET -e local_aborted -r 2 [ 1698.031031] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp1 -d 192.168.206.30@tcp -m GET -e local_aborted -r 2 [ 1698.108295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.30@tcp1 -d 192.168.206.30@tcp1 -m GET -e local_aborted -r 2 [ 1698.158183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.30@tcp 192.168.206.130@tcp [ 1698.189271] LNet: There was an unexpected network error while writing to 192.168.206.130: rc = -22 [ 1698.557229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1698.597860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1698.652233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1698.696839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1698.732991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1698.793608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1698.842382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1698.905876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1698.925822] LNet: 88804:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1698.956524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1698.997359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1699.052394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1699.100857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1699.141144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1699.176992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1699.741739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp -d 192.168.206.130@tcp -m GET -e local_no_route -r 2 [ 1699.785141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.30@tcp -d 192.168.206.130@tcp1 -m GET -e local_no_route -r 2 [ 1699.822557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp -d 192.168.206.30@tcp -m GET -e local_no_route -r 2 [ 1699.856303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.30@tcp -d 192.168.206.30@tcp1 -m GET -e local_no_route -r 2 [ 1699.891462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.30@tcp1 -d 192.168.206.130@tcp -m GET -e local_no_route -r 2 [ 1699.932307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.30@tcp1 -d 192.168.206.130@tcp1 -m GET -e local_no_route -r 2 [ 1699.970767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp1 -d 192.168.206.30@tcp -m GET -e local_no_route -r 2 [ 1700.002757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp1 -d 192.168.206.30@tcp1 -m GET -e local_no_route -r 2 [ 1700.055466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.30@tcp 192.168.206.130@tcp [ 1700.342090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1700.374024] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1700.412155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1700.447211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1700.501786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1700.541485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1700.575799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1700.603200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1700.633172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1700.671692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1700.702727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1700.725962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1700.758931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1700.791432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1701.285660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp -d 192.168.206.130@tcp -m GET -e local_timeout -r 2 [ 1701.318522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp -d 192.168.206.130@tcp1 -m GET -e local_timeout -r 2 [ 1701.350206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp -d 192.168.206.30@tcp -m GET -e local_timeout -r 2 [ 1701.384967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.30@tcp -d 192.168.206.30@tcp1 -m GET -e local_timeout -r 2 [ 1701.425679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.30@tcp1 -d 192.168.206.130@tcp -m GET -e local_timeout -r 2 [ 1701.477280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp1 -d 192.168.206.130@tcp1 -m GET -e local_timeout -r 2 [ 1701.510571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp1 -d 192.168.206.30@tcp -m GET -e local_timeout -r 2 [ 1701.564800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp1 -d 192.168.206.30@tcp1 -m GET -e local_timeout -r 2 [ 1701.602809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.30@tcp 192.168.206.130@tcp [ 1701.627112] LNet: There was an unexpected network error while writing to 192.168.206.130: rc = -22 [ 1701.638675] LNet: Skipped 1 previous similar message [ 1701.864350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1701.896932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1701.924319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1701.953985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1701.984400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1702.021724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1702.047297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1702.077289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1702.093750] LNet: 89045:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1702.098468] LNet: 89045:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1702.113344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1702.143042] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1702.174939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1702.205990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1702.228627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1702.256326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1702.656743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.30@tcp -d 192.168.206.130@tcp -m GET -e remote_dropped -r 2 [ 1702.698147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.30@tcp -d 192.168.206.130@tcp1 -m GET -e remote_dropped -r 2 [ 1702.756960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp -d 192.168.206.30@tcp -m GET -e remote_dropped -r 2 [ 1702.796886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.30@tcp -d 192.168.206.30@tcp1 -m GET -e remote_dropped -r 2 [ 1702.840147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.30@tcp1 -d 192.168.206.130@tcp -m GET -e remote_dropped -r 2 [ 1702.877311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.30@tcp1 -d 192.168.206.130@tcp1 -m GET -e remote_dropped -r 2 [ 1702.918599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.30@tcp1 -d 192.168.206.30@tcp -m GET -e remote_dropped -r 2 [ 1702.952738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.30@tcp1 -d 192.168.206.30@tcp1 -m GET -e remote_dropped -r 2 [ 1702.992430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.30@tcp 192.168.206.130@tcp [ 1703.019287] LNet: 1 peer NIs in recovery (showing 1): 192.168.206.130@tcp [ 1703.218972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1703.264338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1703.310078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1703.361903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1703.388151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1703.408834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1703.435894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1703.461300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1703.493864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1703.523682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1703.558351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1703.598500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1703.649038] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1703.681487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1709.130512] LNet: 89243:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1709.137354] LNetError: 89243:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1709.146162] LNet: Removed LNI 192.168.206.30@tcp [ 1709.163518] LNet: Removed LNI 192.168.206.30@tcp1 [ 1709.737126] Key type .llcrypt unregistered [ 1709.749663] Key type ._llcrypt unregistered [ 1726.468905] Key type ._llcrypt registered [ 1726.479896] Key type .llcrypt registered [ 1727.747587] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 17:08:08 (1787864888) [ 1728.311455] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1728.338559] alg: No test for adler32 (adler32-zlib) [ 1729.266820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1729.326949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1729.375472] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 1730.160532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1730.214790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.206.30@tcp [ 1730.234568] LNetError: 90222:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.206.30@tcp to recovery [ 1730.335503] LNet: 1 local NIs in recovery (showing 1): 192.168.206.30@tcp [ 1730.336836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.206.30@tcp [ 1730.433204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1736.276148] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 17:08:17 (1787864897) [ 1736.417389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1736.447796] LNet: 90810:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1736.473303] LNetError: 90810:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1736.493651] LNet: Removed LNI 192.168.206.30@tcp [ 1736.529067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1736.599326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1736.626352] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 1737.148095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1737.171761] LNet: Added LNI 192.168.206.30@tcp1 [8/256/0/180] [ 1737.967426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.206.30@tcp [ 1737.981900] LNetError: 91079:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.206.30@tcp to recovery [ 1738.603631] LNet: There was an unexpected network error while writing to 192.168.206.30: rc = -22 [ 1739.182424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.206.30@tcp [ 1746.584809] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 17:08:27 (1787864907) [ 1747.913244] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg630-server [ 1749.139632] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 17:08:30 (1787864910) [ 1750.508310] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg630-server [ 1751.873489] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 17:08:33 (1787864913) [ 1753.227936] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg630-server [ 1754.561842] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 17:08:35 (1787864915) [ 1754.832674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1754.856490] LNet: 92710:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1754.869072] LNetError: 92710:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1754.876482] LNet: Removed LNI 192.168.206.30@tcp [ 1754.890298] LNet: Removed LNI 192.168.206.30@tcp1 [ 1754.933754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1754.991508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1755.018959] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 1755.629405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1755.693416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1755.744984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1755.803309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1755.838842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1755.875247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1761.162639] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 17:08:42 (1787864922) [ 1761.306840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1761.342265] LNet: 93435:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1761.353490] LNetError: 93435:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1761.358833] LNetError: 93435:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1761.378532] LNet: Removed LNI 192.168.206.30@tcp [ 1761.401829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1761.452178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1761.488464] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 1767.157813] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 17:08:48 (1787864928) [ 1767.324822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1767.346962] LNet: 94139:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1767.350944] LNetError: 94139:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1767.368350] LNet: Removed LNI 192.168.206.30@tcp [ 1767.390653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1767.455854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1767.486104] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 1773.615712] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 17:08:54 (1787864934) [ 1774.888662] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 1776.191791] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 17:08:57 (1787864937) [ 1776.800371] LNet: 95173:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1776.807865] LNetError: 95173:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1776.823513] LNet: Removed LNI 192.168.206.30@tcp [ 1777.462614] Key type .llcrypt unregistered [ 1777.468269] Key type ._llcrypt unregistered [ 1783.550908] Key type ._llcrypt registered [ 1783.555527] Key type .llcrypt registered [ 1783.779543] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1783.795357] alg: No test for adler32 (adler32-zlib) [ 1784.706545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1784.734547] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 1790.807827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.130@tcp [ 1795.123860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.206.130@tcp [ 1795.468310] LNet: 95598:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1795.472890] LNetError: 95598:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1795.486625] LNet: Removed LNI 192.168.206.30@tcp [ 1795.699511] Key type .llcrypt unregistered [ 1795.703397] Key type ._llcrypt unregistered [ 1807.383970] Key type ._llcrypt registered [ 1807.386458] Key type .llcrypt registered [ 1808.862982] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 1810.443449] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 1812.869883] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 17:09:33 (1787864973) [ 1814.460470] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 [ 1816.697120] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 17:09:37 (1787864977) [ 1819.066574] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg630-server [ 1820.971343] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 17:09:41 (1787864981) [ 1822.676037] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg630-server [ 1824.457322] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 17:09:45 (1787864985) [ 1825.631678] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1825.642839] alg: No test for adler32 (adler32-zlib) [ 1826.824807] Lustre: Lustre: Build Version: 2.17.57_106_g3758560 [ 1827.058673] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 1828.783951] Key type lgssc registered [ 1830.122871] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1849.429889] Key type lgssc unregistered [ 1849.689548] LNet: 98776:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1849.698225] LNetError: 98776:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1849.721813] LNet: Removed LNI 192.168.206.30@tcp [ 1850.400467] Key type .llcrypt unregistered [ 1850.408201] Key type ._llcrypt unregistered [ 1864.571479] Key type ._llcrypt registered [ 1864.573276] Key type .llcrypt registered [ 1865.796682] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 17:10:26 (1787865026) [ 1866.390861] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1866.405690] alg: No test for adler32 (adler32-zlib) [ 1867.356745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1867.438447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1867.491073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1867.549786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1867.588194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1873.772504] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 17:10:35 (1787865035) [ 1873.993882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1874.008891] LNet: 100192:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1874.036656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1874.106857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1874.124399] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 1874.907587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1874.924639] LNet: 100336:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1874.929508] LNetError: 100336:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1874.944487] LNet: Removed LNI 192.168.206.30@tcp [ 1874.966592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1875.014091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1875.048192] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 1875.519118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1875.542487] LNetError: 100364:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1875.559368] LNet: Removed LNI 192.168.206.30@tcp [ 1875.576398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1875.639985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1875.661572] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 1882.343370] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 17:10:43 (1787865043) [ 1882.533384] LNet: 100955:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1882.539706] LNet: 100955:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1882.549666] LNetError: 100955:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1882.573769] LNet: Removed LNI 192.168.206.30@tcp [ 1883.211592] Key type .llcrypt unregistered [ 1883.213348] Key type ._llcrypt unregistered [ 1889.038648] Key type ._llcrypt registered [ 1889.042658] Key type .llcrypt registered [ 1889.177231] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1889.194047] alg: No test for adler32 (adler32-zlib) [ 1890.104478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1890.175029] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 1890.177957] LNet: *** cfs_fail_loc=e003, val=100*** [ 1890.843971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.30@tcp [ 1890.869906] LNet: *** cfs_fail_loc=e003, val=100*** [ 1890.876045] LNet: Skipped 2 previous similar messages [ 1890.889229] Ping reply received with mapping: [ 1890.893980] found metadata reply NI[192.168.206.30@tcp] [ 1890.897328] 1e ce a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 1890.901807] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 1890.906245] 5a 5a 5a 5a 5a 5a 5a 5a [ 1891.079797] LNet: 101356:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1891.092860] LNetError: 101356:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1891.113799] LNet: Removed LNI 192.168.206.30@tcp [ 1891.833550] Key type .llcrypt unregistered [ 1891.837114] Key type ._llcrypt unregistered [ 1901.608373] Key type ._llcrypt registered [ 1901.611221] Key type .llcrypt registered [ 1902.929369] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 17:11:03 (1787865063) [ 1903.571388] Key type .llcrypt unregistered [ 1903.575578] Key type ._llcrypt unregistered [ 1909.453983] Key type ._llcrypt registered [ 1909.456300] Key type .llcrypt registered [ 1909.643192] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1909.663590] alg: No test for adler32 (adler32-zlib) [ 1910.819147] LNetError: 102414:0:(api-ni.c:2850:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 1911.326517] Key type .llcrypt unregistered [ 1911.330075] Key type ._llcrypt unregistered [ 1917.135522] Key type ._llcrypt registered [ 1917.137519] Key type .llcrypt registered [ 1918.475781] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 17:11:19 (1787865079) [ 1919.022285] Key type .llcrypt unregistered [ 1919.024786] Key type ._llcrypt unregistered [ 1924.633976] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1924.639236] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1927.168222] Key type ._llcrypt registered [ 1927.169975] Key type .llcrypt registered [ 1927.385860] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1927.402701] alg: No test for adler32 (adler32-zlib) [ 1928.367787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1928.446427] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1937.086763] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 17:11:38 (1787865098) [ 1937.278983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1937.299839] LNet: 104205:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1937.306958] LNetError: 104205:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1937.322921] LNet: Removed LNI 10.1.2.3@tcp [ 1937.339455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1938.029709] Key type .llcrypt unregistered [ 1938.033953] Key type ._llcrypt unregistered [ 1943.590896] Key type ._llcrypt registered [ 1943.595257] Key type .llcrypt registered [ 1943.819668] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1943.834894] alg: No test for adler32 (adler32-zlib) [ 1944.731464] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 1944.795738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.30@tcp [ 1944.902986] LNet: 104497:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1944.914586] LNetError: 104497:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1944.938409] LNet: Removed LNI 192.168.206.30@tcp [ 1945.510738] Key type .llcrypt unregistered [ 1945.513518] Key type ._llcrypt unregistered [ 1954.981844] Key type ._llcrypt registered [ 1954.984547] Key type .llcrypt registered [ 1956.109515] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 17:11:57 (1787865117) [ 1956.989455] Key type .llcrypt unregistered [ 1956.993486] Key type ._llcrypt unregistered [ 1962.720454] Key type ._llcrypt registered [ 1962.722269] Key type .llcrypt registered [ 1963.066320] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1963.097606] alg: No test for adler32 (adler32-zlib) [ 1968.342872] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 17:12:09 (1787865129) [ 1968.475365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1968.516375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1968.589118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1968.616255] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 1969.131670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1969.161302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 1969.194822] LNet: Removed LNI 192.168.206.30@tcp [ 1969.418846] LNet: 106252:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1969.725532] Key type .llcrypt unregistered [ 1969.730440] Key type ._llcrypt unregistered [ 1978.311440] Key type ._llcrypt registered [ 1978.313630] Key type .llcrypt registered [ 1979.172133] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 17:12:20 (1787865140) [ 1979.511750] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1979.524958] alg: No test for adler32 (adler32-zlib) [ 1980.399100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1980.444552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1980.462623] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 1981.003109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.30@tcp [ 1981.043419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.206.30@tcp [ 1981.068212] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 1981.071410] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 1981.074383] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1981, 0, 0 [ 1981.079624] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 1981.082850] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 1981.087043] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 1981.090102] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1981, 0, 0 [ 1981.094975] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 1981.097840] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1981.101727] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 1981.105668] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1981, 0, 2031 [ 1981.109731] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 1981.112671] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1981.117483] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 1981.121583] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1981, 0, 2031 [ 1981.129895] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 1986.169599] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 17:12:27 (1787865147) [ 1986.534785] LNet: 107791:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1986.539598] LNetError: 107791:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1986.552508] LNet: Removed LNI 192.168.206.30@tcp [ 1987.046763] Key type .llcrypt unregistered [ 1987.053072] Key type ._llcrypt unregistered [ 1992.585532] Key type ._llcrypt registered [ 1992.587686] Key type .llcrypt registered [ 1992.759256] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1992.773249] alg: No test for adler32 (adler32-zlib) [ 1993.660120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1993.680112] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 2000.216220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.130@tcp [ 2007.715986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2007.777783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2007.808169] LNet: Added LNI 192.168.206.30@tcp1 [8/256/0/180] [ 2008.859743] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2009.068086] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2009.071583] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2009.075046] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2011.362281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 2011.389644] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 2022.366836] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 17:13:02 (1787865182) [ 2022.488039] LNet: 109287:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2022.504440] LNetError: 109287:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2022.510358] LNet: Removed LNI 192.168.206.30@tcp [ 2023.519423] LNet: Removed LNI 192.168.206.30@tcp1 [ 2023.522340] LNetError: 109287:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2023.528926] LNetError: 109287:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2023.558177] LNet: Removed LNI 10.1.2.3@tcp99 [ 2024.217609] Key type .llcrypt unregistered [ 2024.224095] Key type ._llcrypt unregistered [ 2031.383560] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2031.390541] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2034.063263] Key type ._llcrypt registered [ 2034.068651] Key type .llcrypt registered [ 2034.301240] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2034.314961] alg: No test for adler32 (adler32-zlib) [ 2035.247546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2035.319242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2035.337340] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 2036.125802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2036.149131] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2037.940700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 2037.976304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2038.027658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 2038.054321] LNetError: 110045:0:(peer.c:2362:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 2038.098509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2038.145588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.30@tcp [ 2038.537338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2038.638677] LNet: 110065:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2038.652645] LNetError: 110065:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2038.659163] LNet: Removed LNI 10.1.2.3@tcp [ 2039.718743] LNet: Removed LNI 192.168.206.30@tcp [ 2040.368086] Key type .llcrypt unregistered [ 2040.374406] Key type ._llcrypt unregistered [ 2052.477606] Key type ._llcrypt registered [ 2052.479131] Key type .llcrypt registered [ 2053.934370] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 17:13:34 (1787865214) [ 2054.526367] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2054.570638] alg: No test for adler32 (adler32-zlib) [ 2055.487119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2055.569775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2055.594902] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 2056.518418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.30@tcp [ 2056.648594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg630-client@tcp [ 2062.673328] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 17:13:43 (1787865223) [ 2062.795276] LNet: 111598:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2062.824343] LNetError: 111598:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2062.844095] LNet: Removed LNI 192.168.206.30@tcp [ 2063.543492] Key type .llcrypt unregistered [ 2063.546682] Key type ._llcrypt unregistered [ 2069.312616] Key type ._llcrypt registered [ 2069.319278] Key type .llcrypt registered [ 2069.507948] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2069.522690] alg: No test for adler32 (adler32-zlib) [ 2070.401228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 2070.426850] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 2070.453926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 2076.313403] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 17:13:57 (1787865237) [ 2077.795117] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 2079.416247] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 17:14:00 (1787865240) [ 2080.729179] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 2082.301718] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 17:14:03 (1787865243) [ 2082.374595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2082.394592] LNet: 113030:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2082.399663] LNetError: 113030:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2082.420533] LNet: Removed LNI 192.168.206.30@tcp [ 2082.442463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2082.526741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2082.559630] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 2083.249415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2083.295066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2089.002650] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 17:14:10 (1787865250) [ 2089.107496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2089.133768] LNet: 113740:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2089.145735] LNetError: 113740:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2089.182105] LNet: Removed LNI 192.168.206.30@tcp [ 2089.217315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2089.276732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 2089.309586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2089.939703] Key type .llcrypt unregistered [ 2089.944512] Key type ._llcrypt unregistered [ 2102.192282] Key type ._llcrypt registered [ 2102.201784] Key type .llcrypt registered [ 2104.148777] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 17:14:24 (1787865264) [ 2104.630302] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2104.668071] alg: No test for adler32 (adler32-zlib) [ 2105.658798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2105.755101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2105.780606] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 2106.796911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2106.911596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.30@tcp [ 2112.495254] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 17:14:33 (1787865273) [ 2112.729643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2112.759604] LNet: 115320:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2112.770712] LNetError: 115320:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2112.784574] LNet: Removed LNI 192.168.206.30@tcp [ 2112.831862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2112.922967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 2112.962321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 2118.112367] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 17:14:39 (1787865279) [ 2118.514895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2118.536255] LNet: 115901:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2118.555781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2118.641870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2119.089858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2119.129509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2119.194487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 2119.603090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2119.625757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2119.677854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2119.695670] LNet: 115934:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2119.712939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2119.756967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2120.035810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2120.070216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2120.126307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 2120.432758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2120.457976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2125.757386] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 17:14:47 (1787865287) [ 2125.927547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2125.953871] LNet: 116535:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2125.974842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2126.034973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2126.064803] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 2126.775403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2126.795568] LNet: Added LNI 192.168.206.30@tcp2 [8/256/0/180] [ 2127.455407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2127.500879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.30@tcp [ 2127.723635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2127.745901] LNetError: 116814:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2127.753058] LNet: Removed LNI 192.168.206.30@tcp [ 2127.764444] LNet: Removed LNI 192.168.206.30@tcp2 [ 2127.794738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2127.840972] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 2127.847705] LNet: Added LNI 192.168.206.30@tcp2 [8/256/0/180] [ 2127.856569] LNet: 116820:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2128.366272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.30@tcp [ 2128.517699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2129.569243] LNet: Removed LNI 192.168.206.30@tcp [ 2129.571131] LNetError: 116857:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2129.575707] LNetError: 116857:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 2129.587196] LNet: Removed LNI 192.168.206.30@tcp2 [ 2129.624706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2129.694294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2129.732961] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 2130.292515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2130.320548] LNet: Added LNI 192.168.206.30@tcp2 [8/256/0/180] [ 2130.948931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.30@tcp [ 2130.990518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2131.030844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.30@tcp [ 2131.198269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2131.211932] LNet: 117143:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2131.218456] LNet: 117143:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 2131.224992] LNet: Removed LNI 192.168.206.30@tcp [ 2132.263770] LNet: Removed LNI 192.168.206.30@tcp2 [ 2132.296865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2132.340455] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 2132.346382] LNet: Added LNI 192.168.206.30@tcp2 [8/256/0/180] [ 2132.354838] LNet: 117149:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2132.837171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.30@tcp [ 2133.023803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2133.040706] LNetError: 117186:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2133.045161] LNetError: 117186:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 2133.049389] LNet: Removed LNI 192.168.206.30@tcp [ 2134.118536] LNet: Removed LNI 192.168.206.30@tcp2 [ 2134.145408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2134.202394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2134.225323] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 2134.753875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2134.770765] LNet: Added LNI 192.168.206.30@tcp2 [8/256/0/180] [ 2135.458752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2135.521593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.30@tcp [ 2135.695923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2136.735327] LNet: Removed LNI 192.168.206.30@tcp [ 2136.747335] LNet: Removed LNI 192.168.206.30@tcp2 [ 2136.779362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2136.822606] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 2136.827684] LNet: Added LNI 192.168.206.30@tcp2 [8/256/0/180] [ 2136.831722] LNet: 117471:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2137.371699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.30@tcp [ 2137.535735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2137.553578] LNetError: 117508:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2137.558628] LNetError: 117508:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 2137.564655] LNet: Removed LNI 192.168.206.30@tcp [ 2137.580939] LNet: Removed LNI 192.168.206.30@tcp2 [ 2137.602717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2137.659133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2137.683027] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 2138.291459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2138.308690] LNet: Added LNI 192.168.206.30@tcp2 [8/256/0/180] [ 2139.180669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.30@tcp [ 2139.212683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2139.259947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.30@tcp [ 2139.457537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2139.477538] LNet: 117793:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2139.487948] LNet: 117793:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 2140.511283] LNet: Removed LNI 192.168.206.30@tcp [ 2141.540727] LNet: Removed LNI 192.168.206.30@tcp2 [ 2141.564068] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2141.630890] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 2141.641370] LNet: Added LNI 192.168.206.30@tcp2 [8/256/0/180] [ 2141.650810] LNet: 117799:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2142.125359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.30@tcp [ 2142.300350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2142.335404] LNet: Removed LNI 192.168.206.30@tcp [ 2143.404928] LNet: Removed LNI 192.168.206.30@tcp2 [ 2143.430571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2143.483643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2143.504403] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 2144.009043] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2144.034359] LNet: Added LNI 192.168.206.30@tcp2 [8/256/0/180] [ 2144.936541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2144.963922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2145.027357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.30@tcp [ 2145.342185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2146.399144] LNet: Removed LNI 192.168.206.30@tcp [ 2146.401356] LNetError: 118133:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2146.406447] LNetError: 118133:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 6 previous similar messages [ 2146.419731] LNet: Removed LNI 192.168.206.30@tcp2 [ 2146.441852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2146.510513] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 2146.520504] LNet: Added LNI 192.168.206.30@tcp2 [8/256/0/180] [ 2146.529475] LNet: 118139:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2147.074450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.30@tcp [ 2147.374936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2148.447138] LNet: Removed LNI 192.168.206.30@tcp [ 2148.457452] LNet: Removed LNI 192.168.206.30@tcp2 [ 2148.483921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2148.549680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2148.595202] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 2149.231717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2149.250096] LNet: Added LNI 192.168.206.30@tcp2 [8/256/0/180] [ 2150.007258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.30@tcp [ 2150.038777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2150.072737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2150.128253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.30@tcp [ 2150.410837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2151.455183] LNet: Removed LNI 192.168.206.30@tcp [ 2152.485741] LNet: Removed LNI 192.168.206.30@tcp2 [ 2152.516470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2152.584079] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 2152.588909] LNet: Added LNI 192.168.206.30@tcp2 [8/256/0/180] [ 2152.996959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.30@tcp [ 2159.220674] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 2160.692260] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 17:15:21 (1787865321) [ 2161.018942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2161.048426] LNet: 119258:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2161.053827] LNet: 119258:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 2161.071527] LNet: Removed LNI 192.168.206.30@tcp [ 2162.161194] LNet: Removed LNI 192.168.206.30@tcp2 [ 2162.194839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2162.509537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2162.533826] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 2163.181488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2163.198956] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2164.021616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2164.040773] LNetError: 119531:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2164.044996] LNetError: 119531:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 6 previous similar messages [ 2164.048630] LNet: Removed LNI 10.1.2.3@tcp [ 2164.057728] LNet: Removed LNI 192.168.206.30@tcp [ 2164.082772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2164.149551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 2164.167546] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 2164.174798] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2164.576163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 2164.609393] LNet: Removed LNI 192.168.206.30@tcp [ 2164.622710] LNet: Removed LNI 10.1.2.3@tcp [ 2170.522861] Lustre: DEBUG MARKER: == sanity-lnet test 475: per-operation latency stats populate and reset ========================================================== 17:15:31 (1787865331) [ 2170.653341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2170.708202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2170.804525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2170.841646] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 2171.704425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2171.749861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.30@tcp [ 2171.793373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.30@tcp [ 2171.834936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.30@tcp [ 2171.874430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.30@tcp [ 2171.918306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.30@tcp [ 2171.971856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.30@tcp [ 2172.036697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.30@tcp [ 2172.079879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.30@tcp [ 2172.128615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.30@tcp [ 2172.176629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.30@tcp [ 2172.316507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2178.178351] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 17:15:39 (1787865339) [ 2178.257528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2179.303812] LNet: Removed LNI 192.168.206.30@tcp [ 2179.338852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2179.475723] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2179.644847] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2179.662606] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2179.678105] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2182.036891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2182.074378] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 2182.761931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2182.808715] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2184.416555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.30@tcp [ 2184.449512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 2184.505687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.206.30@tcp -m PUT -e local_timeout -r 1 [ 2184.552606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 2184.598759] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2184.604888] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2184.909665] LNetError: 120940:0:(lib-move.c:4801:lnet_parse()) 192.168.206.30@tcp, src 192.168.206.30@tcp: Dropping PUT (error -108 looking up sender) [ 2185.960269] LNet: Removed LNI 10.1.2.3@tcp [ 2186.981685] LNet: Removed LNI 192.168.206.30@tcp [ 2187.371870] Key type .llcrypt unregistered [ 2187.377804] Key type ._llcrypt unregistered [ 2199.080565] Key type ._llcrypt registered [ 2199.086288] Key type .llcrypt registered [ 2200.726386] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 17:16:01 (1787865361) [ 2201.348938] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2201.374967] alg: No test for adler32 (adler32-zlib) [ 2202.408417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2202.927340] LNet: 122272:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2203.543569] Key type .llcrypt unregistered [ 2203.545633] Key type ._llcrypt unregistered [ 2210.073775] Key type ._llcrypt registered [ 2210.077604] Key type .llcrypt registered [ 2210.264497] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2210.275216] alg: No test for adler32 (adler32-zlib) [ 2211.296932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2211.339698] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 2218.165245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.130@tcp [ 2218.379839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 2219.068870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.206.130@tcp [ 2219.376940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2219.451757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2219.497244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2219.542450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2219.551196] LNet: 1 peer NIs in recovery (showing 1): 192.168.206.130@tcp [ 2219.587347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2219.627376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2219.671660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2219.706896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2219.728745] LNet: 122725:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2219.747385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2219.781482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2219.816653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2219.856190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2219.886701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2219.921351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2220.695300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2220.898435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2220.926511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2220.966889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2220.987646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2221.018873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2221.045289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2221.073458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2221.102377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2221.130136] LNet: 122843:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2221.170600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2221.205323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2221.235875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2221.267842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2221.304901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2221.332696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2221.849209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.206.130@tcp [ 2222.091445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2222.123113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2222.151517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2222.176411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2222.207185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2222.238906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2222.258694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2222.284480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2222.302499] LNet: 122960:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2222.322475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2222.358097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2222.385543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2222.423753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2222.459615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2222.490622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2223.024119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2223.206847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2223.241492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2223.276269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2223.322336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2223.366451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2223.391948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2223.436926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2223.471492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2223.497516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2223.528148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2223.567293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2223.595926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2223.621212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2223.657371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2223.997912] LNet: 123127:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2224.004062] LNetError: 123127:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2224.025322] LNet: Removed LNI 192.168.206.30@tcp [ 2224.450696] Key type .llcrypt unregistered [ 2224.453753] Key type ._llcrypt unregistered [ 2241.366949] Key type ._llcrypt registered [ 2241.370855] Key type .llcrypt registered [ 2242.881167] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 17:16:43 (1787865403) [ 2243.399942] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2243.433648] alg: No test for adler32 (adler32-zlib) [ 2244.349291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2244.846122] LNet: 124014:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2245.702626] Key type .llcrypt unregistered [ 2245.704319] Key type ._llcrypt unregistered [ 2252.817754] Key type ._llcrypt registered [ 2252.821098] Key type .llcrypt registered [ 2253.020733] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2253.045901] alg: No test for adler32 (adler32-zlib) [ 2254.053613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2254.080094] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 2262.586619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.130@tcp [ 2271.188082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2271.276225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2271.304776] LNet: Added LNI 192.168.206.30@tcp1 [8/256/0/180] [ 2272.414615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 2273.062838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.206.130@tcp [ 2273.292688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2273.329522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2273.358502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2273.386679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2273.435140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2273.463843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2273.490376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2273.531363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2273.561416] LNet: 124670:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2273.571264] LNet: 1 peer NIs in recovery (showing 1): 192.168.206.130@tcp [ 2273.593486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2273.629825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2273.664889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2273.703491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2273.743608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2273.772799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2274.257360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2274.457595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2274.486565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2274.515782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2274.539471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2274.562771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2274.585160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2274.618292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2274.664807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2274.683291] LNet: 124787:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2274.709054] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2274.735693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2274.770073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2274.804126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2274.842043] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2274.893889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2275.513508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.206.130@tcp [ 2275.750358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2275.774767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2275.799226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2275.824430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2275.867547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2275.912796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2275.960058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2276.009575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2276.027473] LNet: 124905:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2276.045453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2276.087230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2276.120462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2276.148263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2276.177352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2276.212375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2276.862726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2277.120962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2277.148637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2277.193953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2277.229850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2277.255854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2277.277404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2277.304183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2277.334517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2277.368949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2277.395714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2277.441841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2277.483754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2277.516726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2277.612991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2282.636229] LNet: 125107:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2282.645368] LNetError: 125107:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2282.655863] LNet: Removed LNI 192.168.206.30@tcp [ 2282.669134] LNet: Removed LNI 192.168.206.30@tcp1 [ 2283.123806] Key type .llcrypt unregistered [ 2283.126321] Key type ._llcrypt unregistered [ 2299.275125] Key type ._llcrypt registered [ 2299.278526] Key type .llcrypt registered [ 2300.558682] Lustre: DEBUG MARKER: == sanity-lnet test 525: keep reachable peer on local net delete ========================================================== 17:17:41 (1787865461) [ 2300.953226] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2300.975369] alg: No test for adler32 (adler32-zlib) [ 2301.868962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2301.939154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2301.965671] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 2302.646693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2302.665858] LNet: Added LNI 192.168.206.30@tcp2 [8/256/0/180] [ 2303.350791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.30@tcp [ 2303.410903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp [ 2304.481167] LNet: Removed LNI 192.168.206.30@tcp [ 2310.132327] Lustre: DEBUG MARKER: == sanity-lnet test 550: DD-off: keep route up when gw reachable via another local NI ========================================================== 17:17:51 (1787865471) [ 2311.417629] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_550 Need at least 2 local interfaces for multi-rail [ 2312.875387] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 17:17:54 (1787865474) [ 2318.424959] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 17:17:59 (1787865479) [ 2324.916344] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 17:18:05 (1787865485) [ 2331.494494] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 17:18:12 (1787865492) [ 2338.951518] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 17:18:19 (1787865499) [ 2346.868953] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 17:18:27 (1787865507) [ 2354.278540] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 17:18:35 (1787865515) [ 2360.696673] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 17:18:41 (1787865521) [ 2369.818064] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 17:18:50 (1787865530) [ 2377.896826] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 17:18:58 (1787865538) [ 2384.732637] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 17:19:05 (1787865545) [ 2390.577640] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 17:19:11 (1787865551) [ 2397.199161] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 17:19:18 (1787865558) [ 2404.872632] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 17:19:25 (1787865565) [ 2411.220794] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 17:19:32 (1787865572) [ 2418.184724] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 17:19:39 (1787865579) [ 2424.421745] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 17:19:45 (1787865585) [ 2431.760733] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 17:19:52 (1787865592) [ 2431.835948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2431.872107] LNet: 136794:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2431.885784] LNetError: 136794:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2431.905119] LNet: Removed LNI 192.168.206.30@tcp2 [ 2431.938174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2432.053409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2432.070072] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 2432.922487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2432.942657] LNet: 136942:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2432.949727] LNetError: 136942:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2432.977304] LNet: Removed LNI 192.168.206.30@tcp [ 2433.001889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2433.065185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2433.084365] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 2438.486189] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 17:19:59 (1787865599) [ 2445.352780] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 17:20:06 (1787865606) [ 2451.318339] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 17:20:12 (1787865612) [ 2457.761487] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 17:20:19 (1787865619) [ 2462.718710] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 17:20:24 (1787865624) [ 2469.787282] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 17:20:30 (1787865630) [ 2476.086801] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 17:20:37 (1787865637) [ 2485.006787] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 17:20:45 (1787865645) [ 2493.622149] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 17:20:54 (1787865654) [ 2501.808234] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 17:21:02 (1787865662) [ 2508.412710] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 17:21:09 (1787865669) [ 2514.811826] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 17:21:15 (1787865675) [ 2521.602509] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 2271 sec ========== 17:21:22 (1787865682) [ 2521.946636] LNet: 144408:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2521.952768] LNetError: 144408:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2521.968675] LNet: Removed LNI 192.168.206.30@tcp [ 2522.768780] Key type .llcrypt unregistered [ 2522.773272] Key type ._llcrypt unregistered [ 2536.200945] Key type ._llcrypt registered [ 2536.204245] Key type .llcrypt registered [ 2537.188323] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2537.207295] alg: No test for adler32 (adler32-zlib) [ 2538.605394] Lustre: Lustre: Build Version: 2.17.57_106_g3758560 [ 2539.004038] LNet: Added LNI 192.168.206.30@tcp [8/256/0/180] [ 2540.815554] Key type lgssc registered [ 2542.643588] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2614.820780] Lustre: Mounted lustre-client - version 2.17.57_106_g3758560 [ 2621.553737] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2632.427626] Lustre: Unmounted lustre-client [ 2662.317687] Key type lgssc unregistered [ 2662.629682] LNet: 147371:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2662.639281] LNetError: 147371:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2662.662819] LNet: Removed LNI 192.168.206.30@tcp [ 2663.657142] Key type .llcrypt unregistered [ 2663.658773] Key type ._llcrypt unregistered