[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 462563721 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001013] APIC: Switch to symmetric I/O mode setup [ 0.003270] x2apic enabled [ 0.005004] Switched APIC routing to physical x2apic. [ 0.006014] kvm-guest: setup PV IPIs [ 0.008803] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009005] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.010019] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.011012] pid_max: default: 32768 minimum: 301 [ 0.013042] LSM: Security Framework initializing [ 0.014049] Yama: becoming mindful. [ 0.016012] SELinux: Initializing. [ 0.017081] *** VALIDATE selinux *** [ 0.025603] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.030144] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.031138] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.032101] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.033100] *** VALIDATE tmpfs *** [ 0.034448] *** VALIDATE proc *** [ 0.036185] *** VALIDATE cgroup *** [ 0.037008] *** VALIDATE cgroup2 *** [ 0.039092] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.040142] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.041007] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.042029] Spectre V2 : User space: Vulnerable [ 0.043010] Speculative Store Bypass: Vulnerable [ 0.046414] debug: unmapping init [mem 0xffffffff99259000-0xffffffff99260fff] [ 0.048156] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.049689] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.050025] ... version: 2 [ 0.051011] ... bit width: 48 [ 0.052010] ... generic registers: 4 [ 0.053013] ... value mask: 0000ffffffffffff [ 0.054015] ... max period: 00007fffffffffff [ 0.055015] ... fixed-purpose events: 3 [ 0.056013] ... event mask: 000000070000000f [ 0.057282] rcu: Hierarchical SRCU implementation. [ 0.059398] smp: Bringing up secondary CPUs ... [ 0.060741] x86: Booting SMP configuration: [ 0.061026] .... node #0, CPUs: #1 #2 #3 [ 0.065013] smp: Brought up 1 node, 4 CPUs [ 0.067018] smpboot: Max logical packages: 1 [ 0.068014] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.142199] node 0 deferred pages initialised in 71ms [ 0.145349] devtmpfs: initialized [ 0.146276] x86/mm: Memory block size: 128MB [ 0.148787] gcov: version magic: 0x41383552 [ 0.150472] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.153091] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.155344] pinctrl core: initialized pinctrl subsystem [ 0.158180] [ 0.159008] ************************************************************* [ 0.162016] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.165013] ** ** [ 0.168014] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.171013] ** ** [ 0.174015] ** This means that this kernel is built to expose internal ** [ 0.177020] ** IOMMU data structures, which may compromise security on ** [ 0.180022] ** your system. ** [ 0.183020] ** ** [ 0.185014] ** If you see this message and you are not debugging the ** [ 0.187017] ** kernel, report this immediately to your vendor! ** [ 0.189013] ** ** [ 0.191017] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.193014] ************************************************************* [ 0.196744] NET: Registered protocol family 16 [ 0.199448] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.202060] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.204060] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.208085] cpuidle: using governor menu [ 0.209626] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.210451] PCI: Using configuration type 1 for base access [ 0.212140] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.224063] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.225048] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.227140] cryptd: max_cpu_qlen set to 1000 [ 0.229249] ACPI: Added _OSI(Module Device) [ 0.231024] ACPI: Added _OSI(Processor Device) [ 0.232011] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.234014] ACPI: Added _OSI(Processor Aggregator Device) [ 0.240497] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.246600] ACPI: Interpreter enabled [ 0.248066] ACPI: PM: (supports S0 S3 S4 S5) [ 0.250010] ACPI: Using IOAPIC for interrupt routing [ 0.251089] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.255390] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.266000] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.268035] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.270017] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.273071] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.277238] acpiphp: Slot [2] registered [ 0.278098] acpiphp: Slot [5] registered [ 0.279083] acpiphp: Slot [6] registered [ 0.281100] acpiphp: Slot [3] registered [ 0.282119] acpiphp: Slot [4] registered [ 0.283089] acpiphp: Slot [7] registered [ 0.285097] acpiphp: Slot [8] registered [ 0.286132] acpiphp: Slot [9] registered [ 0.288091] acpiphp: Slot [10] registered [ 0.289079] acpiphp: Slot [11] registered [ 0.291105] acpiphp: Slot [12] registered [ 0.292093] acpiphp: Slot [13] registered [ 0.294168] acpiphp: Slot [14] registered [ 0.296179] acpiphp: Slot [15] registered [ 0.297093] acpiphp: Slot [16] registered [ 0.299101] acpiphp: Slot [17] registered [ 0.301102] acpiphp: Slot [18] registered [ 0.303105] acpiphp: Slot [19] registered [ 0.304091] acpiphp: Slot [20] registered [ 0.306121] acpiphp: Slot [21] registered [ 0.308119] acpiphp: Slot [22] registered [ 0.310082] acpiphp: Slot [23] registered [ 0.311103] acpiphp: Slot [24] registered [ 0.313093] acpiphp: Slot [25] registered [ 0.315099] acpiphp: Slot [26] registered [ 0.316129] acpiphp: Slot [27] registered [ 0.318076] acpiphp: Slot [28] registered [ 0.320101] acpiphp: Slot [29] registered [ 0.322091] acpiphp: Slot [30] registered [ 0.324112] acpiphp: Slot [31] registered [ 0.325078] PCI host bridge to bus 0000:00 [ 0.327021] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.329030] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.331026] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.334022] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.337026] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.339074] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.341232] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.344606] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.348838] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.359014] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.363058] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.365019] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.367016] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.370022] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.371598] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.374870] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.377042] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.381012] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.385000] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.392904] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.396000] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.403419] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.410015] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.416013] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.435017] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.446078] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.454017] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.460021] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.477014] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.488407] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.490364] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.492450] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.494369] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.496258] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.502113] iommu: Default domain type: Passthrough [ 0.504735] SCSI subsystem initialized [ 0.507227] ACPI: bus type USB registered [ 0.509129] usbcore: registered new interface driver usbfs [ 0.512085] usbcore: registered new interface driver hub [ 0.514074] usbcore: registered new device driver usb [ 0.516177] pps_core: LinuxPPS API ver. 1 registered [ 0.518013] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.520052] PTP clock support registered [ 0.522105] EDAC MC: Ver: 3.0.0 [ 0.523402] PCI: Using ACPI for IRQ routing [ 0.525737] NetLabel: Initializing [ 0.527011] NetLabel: domain hash size = 128 [ 0.529011] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.531100] NetLabel: unlabeled traffic allowed by default [ 0.533149] vgaarb: loaded [ 0.535053] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.537013] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.543108] clocksource: Switched to clocksource kvm-clock [ 0.652755] VFS: Disk quotas dquot_6.6.0 [ 0.655678] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.658734] *** VALIDATE ramfs *** [ 0.660097] *** VALIDATE hugetlbfs *** [ 0.662756] pnp: PnP ACPI init [ 0.665659] pnp: PnP ACPI: found 6 devices [ 0.685848] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.689440] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.691880] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.694490] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.697024] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.699562] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.701928] NET: Registered protocol family 2 [ 0.704208] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.709093] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.712401] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.718121] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.721853] TCP: Hash tables configured (established 65536 bind 65536) [ 0.725569] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.728899] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.731644] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.734331] NET: Registered protocol family 1 [ 0.736804] RPC: Registered named UNIX socket transport module. [ 0.738784] RPC: Registered udp transport module. [ 0.740240] RPC: Registered tcp transport module. [ 0.742164] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.744222] NET: Registered protocol family 44 [ 0.746152] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.748499] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.753072] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.755170] PCI: CLS 0 bytes, default 64 [ 0.756672] Unpacking initramfs... [ 2.180372] debug: unmapping init [mem 0xffff8afcbcc64000-0xffff8afcbffcffff] [ 2.184360] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.186554] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.188831] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.678312] Initialise system trusted keyrings [ 2.679794] Key type blacklist registered [ 2.681287] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.692189] zbud: loaded [ 2.695674] *** VALIDATE nfs *** [ 2.696854] *** VALIDATE nfs4 *** [ 2.699179] pstore: using deflate compression [ 2.702812] Platform Keyring initialized [ 2.815769] NET: Registered protocol family 38 [ 2.819046] Key type asymmetric registered [ 2.820888] Asymmetric key parser 'x509' registered [ 2.823135] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.826726] io scheduler mq-deadline registered [ 2.828801] io scheduler kyber registered [ 2.830792] io scheduler bfq registered [ 2.833352] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.836676] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.839196] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.841958] ACPI: Power Button [PWRF] [ 2.847873] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.854686] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.864435] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.894362] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.923337] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.930034] Non-volatile memory driver v1.3 [ 2.932056] Linux agpgart interface v0.103 [ 2.968747] virtio_blk virtio1: [vda] 146656 512-byte logical blocks (75.1 MB/71.6 MiB) [ 2.971356] vda: detected capacity change from 0 to 75087872 [ 2.995525] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 2.998270] vdb: detected capacity change from 0 to 1073741824 [ 3.010430] libphy: Fixed MDIO Bus: probed [ 3.016330] usbcore: registered new interface driver usbserial_generic [ 3.019093] usbserial: USB Serial support registered for generic [ 3.022383] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.027443] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.029681] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.032327] mousedev: PS/2 mouse device common for all mice [ 3.035344] rtc_cmos 00:05: RTC can wake from S4 [ 3.038940] rtc_cmos 00:05: registered as rtc0 [ 3.038984] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.040910] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.047155] intel_pstate: CPU model not supported [ 3.050196] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.054031] hid: raw HID events driver (C) Jiri Kosina [ 3.056881] usbcore: registered new interface driver usbhid [ 3.056968] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.058816] usbhid: USB HID core driver [ 3.059110] drop_monitor: Initializing network drop monitor service [ 3.066220] Initializing XFRM netlink socket [ 3.068561] NET: Registered protocol family 10 [ 3.072088] Segment Routing with IPv6 [ 3.073370] NET: Registered protocol family 17 [ 3.075786] mpls_gso: MPLS GSO support [ 3.083897] RAS: Correctable Errors collector initialized. [ 3.086716] AVX version of gcm_enc/dec engaged. [ 3.088573] AES CTR mode by8 optimization enabled [ 3.157287] sched_clock: Marking stable (3157265575, 0)->(4050266512, -893000937) [ 3.161382] registered taskstats version 1 [ 3.164158] Loading compiled-in X.509 certificates [ 3.166331] zswap: loaded using pool lzo/zbud [ 3.189751] Key type big_key registered [ 3.203651] Key type encrypted registered [ 3.205641] ima: No TPM chip found, activating TPM-bypass! [ 3.208210] ima: Allocated hash algorithm: sha1 [ 3.210198] ima: No architecture policies found [ 3.212296] evm: Initialising EVM extended attributes: [ 3.214498] evm: security.selinux [ 3.215885] evm: security.ima [ 3.217220] evm: security.capability [ 3.218906] evm: HMAC attrs: 0x1 [ 3.221261] rtc_cmos 00:05: setting system clock to 2026-08-27 11:20:26 UTC (1787829626) [ 3.227876] debug: unmapping init [mem 0xffffffff9a203000-0xffffffff9a3fffff] [ 3.231594] debug: unmapping init [mem 0xffffffff98f82000-0xffffffff99258fff] [ 3.241159] Write protecting the kernel read-only data: 28672k [ 3.245271] debug: unmapping init [mem 0xffffffff97603000-0xffffffff977fffff] [ 3.248614] debug: unmapping init [mem 0xffffffff97f14000-0xffffffff97ffffff] [ 3.279253] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.289183] systemd[1]: Detected virtualization kvm. [ 3.291184] systemd[1]: Detected architecture x86-64. [ 3.293475] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.321737] systemd[1]: No hostname configured. [ 3.323842] systemd[1]: Set hostname to . [ 3.326322] random: systemd: uninitialized urandom read (16 bytes read) [ 3.329549] systemd[1]: Initializing machine ID from random generator. [ 3.364232] random: ln: uninitialized urandom read (6 bytes read) [ 3.461305] random: systemd: uninitialized urandom read (16 bytes read) [ 3.466972] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 3.474316] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 3.479442] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ OK ] Reached target Timers. [ OK ] Reached target Paths. [ OK ] Reached target Local File Systems. [ OK ] Reached target Slices. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Initrd Root Device. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on Journal Socket. Starting Setup Virtual Console... Starting Create Volatile Files and Directories... [ OK ] Reached target Sockets. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Memstrack Anylazing Service. Starting Apply Kernel Variables... [ OK ] Reached target Local Encrypted Volumes. Starting Journal Service... [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.130302] device-mapper: uevent: version 1.0.3 [ 4.132711] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 4.838165] virtio_net virtio0 ens2: renamed from eth0 [ 4.872496] random: fast init done [ 4.902833] scsi host0: ata_piix [ 4.939816] scsi host1: ata_piix [ 4.941432] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 4.944047] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 9.171184] dracut-initqueue[587]: RTNETLINK answers: File exists [ 9.735848] random: crng init done [ 9.737229] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.042419] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Apply Kernel Variables. Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.151338] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.423810] SELinux: Disabled at runtime. [ 11.486669] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.493863] systemd[1]: Detected virtualization kvm. [ 11.495701] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 11.984660] systemd[1]: initrd-switch-root.service: Succeeded. [ 11.987512] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 11.991884] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 11.995581] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 11.998786] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.006762] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.014281] systemd[1]: Starting Create list of required static device nodes for the current kernel... Starting Create list of required st…ce nodes for the current kernel... Mounting Huge Pages File System... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on udev Control Socket. Mounting Kernel Debug File System... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on RPCbind Server Activation Socket. Starting Remount Root and Kernel File Systems... [ OK ] Reached target rpc_pipefs.target. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Listening on Process Core Dump Socket. Mounting POSIX Message Queue File System... [ OK ] Stopped target Initrd File Systems. [ OK ] Reached target Paths. [ OK ] Reached target RPC Port Mapper. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Created slice system-getty.slice. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Starting Apply Kernel Variables... Activating swap /dev/disk/by-label/SWAP... [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. [FAILED] Failed to start Remount Root and Kernel File Systems. [ 12.334393] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Apply Kernel Variables. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started udev Coldplug all Devices. [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Mounted /mnt. [ 13.174170] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 14.161330] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 14.300288] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 14.771664] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 14.845784] EDAC sbridge: Ver: 1.1.2 [ 17.958941] Key type dns_resolver registered [ 18.827335] NFS: Registering the id_resolver key type [ 18.829953] Key type id_resolver registered [ 18.831588] Key type id_legacy registered [* ] A start job is running for Configur…-only root support (7s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... Starting Login Service... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. [ OK ] Started irqbalance daemon. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... [ OK ] Started Login Service. Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg634-client login: [ 89.056511] libcfs: loading out-of-tree module taints kernel. [ 89.162306] Key type ._llcrypt registered [ 89.166021] Key type .llcrypt registered [ 89.950888] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 89.991166] alg: No test for adler32 (adler32-zlib) [ 91.492785] Lustre: Lustre: Build Version: 2.17.57_64_gb96373c [ 92.026764] LNet: Added LNI 192.168.206.34@tcp [8/256/0/180] [ 93.703191] Key type lgssc registered [ 95.394250] Lustre: Echo OBD driver; http://www.lustre.org/ [ 121.118241] hrtimer: interrupt took 6211072 ns [ 232.835774] Lustre: Mounted lustre-client - version 2.17.57_64_gb96373c [ 238.232429] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 251.400177] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing check_logdir /tmp/testlogs/ [ 256.570944] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing yml_node [ 258.532259] Lustre: lustre-OST0000-osc-ffff8afd107ce000: disconnect after 23s idle [ 260.472987] Lustre: DEBUG MARKER: Client: 2.17.57.64 [ 263.027298] Lustre: DEBUG MARKER: MDS: 2.17.57.64 [ 265.809104] Lustre: DEBUG MARKER: OSS: 2.17.57.64 [ 267.582587] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Thu Aug 27 07:24:49 EDT 2026 [ 282.267758] Lustre: Unmounted lustre-client [ 316.824712] Key type lgssc unregistered [ 317.037075] LNet: 8405:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 317.047655] LNetError: 8405:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 317.068281] LNet: Removed LNI 192.168.206.34@tcp [ 317.676147] Key type .llcrypt unregistered [ 317.677448] Key type ._llcrypt unregistered [ 342.421853] Key type ._llcrypt registered [ 342.425557] Key type .llcrypt registered [ 342.750251] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 342.773546] alg: No test for adler32 (adler32-zlib) [ 343.719642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 343.782558] LNet: Added LNI 192.168.206.34@tcp [8/256/0/180] [ 343.803313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 348.101844] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 348.233645] LNet: 9289:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 348.241989] LNetError: 9289:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 348.272748] LNet: Removed LNI 192.168.206.34@tcp [ 348.950729] Key type .llcrypt unregistered [ 348.953243] Key type ._llcrypt unregistered [ 354.466495] Key type ._llcrypt registered [ 354.468230] Key type .llcrypt registered [ 355.515622] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 07:26:17 (1787829977) [ 355.852162] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 355.867411] alg: No test for adler32 (adler32-zlib) [ 356.812158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 356.893173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 356.929231] LNet: 9710:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 356.958770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 357.015756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 357.031664] LNet: 9717:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 363.310648] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 07:26:25 (1787829985) [ 363.609287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 363.632054] LNet: 10294:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 363.653838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 363.767317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 370.529874] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 07:26:32 (1787829992) [ 370.904946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 370.927545] LNet: 10881:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 370.944028] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 370.994355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 377.897262] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 07:26:39 (1787829999) [ 377.986267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 378.021671] LNet: 11458:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 378.053968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 378.165114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 384.228418] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 07:26:46 (1787830006) [ 384.490792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 384.519079] LNet: 12051:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 384.534880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 384.633329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 385.161704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 392.200292] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 07:26:54 (1787830014) [ 392.300468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 392.337393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 392.459763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 392.876147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 393.399306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 400.130730] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 07:27:02 (1787830022) [ 400.483706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 400.498800] LNet: 13238:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 400.502516] LNet: 13238:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 400.527940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 400.612594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 406.792816] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 07:27:08 (1787830028) [ 407.076207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 407.142526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 407.246231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 407.299514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 407.774908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 407.829553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 408.636369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 408.687074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 409.089339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 409.123159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 409.469865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 409.514482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 410.008861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 410.049220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 410.669049] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 410.704707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 410.981173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 411.019495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 411.536813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 411.576925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 412.052278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 412.094707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 412.532308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 412.574533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 412.877783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 412.923908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 413.323964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 413.368943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 419.903712] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 07:27:21 (1787830041) [ 420.181393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 420.207757] LNet: 14505:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 420.211858] LNet: 14505:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 420.252948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 420.337664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 420.374753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 427.162457] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 07:27:29 (1787830049) [ 427.306056] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 427.357822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 427.459421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 427.505753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 433.391193] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 07:27:35 (1787830055) [ 433.520744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 433.585393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 433.693852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 433.736537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 440.492373] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 07:27:42 (1787830062) [ 440.633586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 440.677326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 440.734525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 440.758974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 447.115850] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 07:27:49 (1787830069) [ 447.181052] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 447.219284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 447.315311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 447.363396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 453.491407] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 07:27:55 (1787830075) [ 453.839328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 453.883546] LNet: 17435:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 453.893483] LNet: 17435:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 453.929250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 454.037563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 454.081087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 460.751708] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 07:28:02 (1787830082) [ 460.999591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 461.032043] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 461.144209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 461.584885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 468.262876] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 07:28:10 (1787830090) [ 468.499537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 468.554032] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 468.728792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 468.896179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 475.616205] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 07:28:17 (1787830097) [ 475.871402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 475.910360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 476.007932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 476.138602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 482.111663] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 07:28:24 (1787830104) [ 482.461969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 482.493700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 482.915994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 483.280206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 489.744748] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 07:28:31 (1787830111) [ 489.892672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 489.962925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 490.032247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 490.098230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 497.639183] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 07:28:39 (1787830119) [ 497.874156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 497.907578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 497.991906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 498.036460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 505.693932] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 07:28:47 (1787830127) [ 505.792987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 505.832991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 505.935734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 513.340836] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 07:28:55 (1787830135) [ 513.464866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 513.503755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 513.593326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 519.304959] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 07:29:01 (1787830141) [ 519.564988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 519.613860] LNet: 22872:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 519.620316] LNet: 22872:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 8 previous similar messages [ 519.725490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 519.828435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 520.494494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 528.000044] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 07:29:09 (1787830149) [ 528.191566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 528.244494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 528.353813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 528.401254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 536.118604] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 07:29:18 (1787830158) [ 536.616520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 536.663403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 536.814263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 536.909833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 545.695766] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 07:29:27 (1787830167) [ 545.780930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 545.849853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 545.986326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 546.042573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 553.832648] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 07:29:35 (1787830175) [ 554.049968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 554.105723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 554.189088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 554.223236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 560.563871] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 07:29:42 (1787830182) [ 560.972974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 561.064870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 561.115798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 561.168943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 561.198259] LNetError: 25822:0:(peer.c:2362:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 561.263854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 561.317797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 568.994608] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 07:29:50 (1787830190) [ 569.570477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 569.659645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 569.838540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.NUoX7P4inm [ 569.980640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.NUoX7P4inm [ 570.056835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.NUoX7P4inm [ 578.921195] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 07:30:00 (1787830200) [ 579.229690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 579.416827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 579.570270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 579.632377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 579.730401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 588.652726] Lustre: DEBUG MARKER: == sanity-lnet test 40: LNetAddPeer merges a pre-existing non-primary peer NID ========================================================== 07:30:10 (1787830210) [ 588.829070] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 588.914521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 589.082773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 589.127965] LNet: Added LNI 192.168.206.34@tcp [8/256/0/180] [ 590.827146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.34@tcp [ 592.144554] Lustre: Lustre: Build Version: 2.17.57_64_gb96373c [ 594.239208] Key type lgssc registered [ 595.435659] Lustre: Echo OBD driver; http://www.lustre.org/ [ 599.396217] LNetError: 27584:0:(lib-move.c:2158:lnet_handle_find_routed_path()) peer 192.168.206.34@tcp99 has no available nets [ 602.240971] Key type lgssc unregistered [ 602.594964] LNetError: 28695:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 602.624304] LNet: Removed LNI 192.168.206.34@tcp [ 603.155560] Key type .llcrypt unregistered [ 603.160531] Key type ._llcrypt unregistered [ 609.250310] Key type ._llcrypt registered [ 609.254196] Key type .llcrypt registered [ 610.397517] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 07:30:32 (1787830232) [ 611.009788] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 611.021908] alg: No test for adler32 (adler32-zlib) [ 611.909558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 611.963413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 613.795635] LNet: Message forwarding will be enabled in 2s [ 616.863525] LNet: Message forwarding enabled [ 617.256983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 617.277480] LNet: Message forwarding will stop in 2s to 6s [ 621.753307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 622.468734] LNet: Message forwarding disabled [ 623.798982] LNet: Message forwarding will be enabled in 2s [ 627.617214] LNet: Message forwarding enabled [ 630.759330] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 07:30:52 (1787830252) [ 630.974476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 631.018217] LNet: 30102:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 632.135824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 632.203323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 638.260285] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 07:31:00 (1787830260) [ 638.438489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 638.463239] LNet: 30682:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 638.510447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 638.548602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 638.568253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 638.589992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 638.618995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 638.649306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 638.672601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 638.690982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 638.714941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 638.760659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 638.794439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 638.819413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 638.851935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 638.910468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 638.950239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 638.991846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 639.018173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 639.091836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 639.159337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 639.216641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 639.230556] LNetError: 30726:0:(router.c:625:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 646.076218] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 07:31:07 (1787830267) [ 646.183628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 646.208875] LNet: 31291:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 646.252470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 646.367389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 652.815414] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 07:31:14 (1787830274) [ 654.613705] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg634-server [ 656.131623] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 07:31:18 (1787830278) [ 656.509506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 656.544356] LNet: 32228:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 656.573815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 656.615686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 656.638748] LNet: Added LNI 192.168.206.34@tcp [8/256/0/180] [ 657.348730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.206.[35-39]@tcp [ 657.390748] LNet: 32361:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 657.442738] LNetError: 32232:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.206.36@tcp is being used as a gateway but routing feature is not turned on [ 657.465644] LNetError: 32232:0:(lib-lnet.h:1280:lnet_set_route_aliveness()) route to tcp1 through 192.168.206.36@tcp has gone from up to down [ 658.466261] LNetError: 32232:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.206.38@tcp is being used as a gateway but routing feature is not turned on [ 658.479822] LNetError: 32232:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 659.499436] LNetError: 32232:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.206.36@tcp is being used as a gateway but routing feature is not turned on [ 659.515485] LNetError: 32232:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 4 previous similar messages [ 661.542833] LNetError: 32232:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.206.36@tcp is being used as a gateway but routing feature is not turned on [ 661.559260] LNetError: 32232:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 664.202945] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 07:31:25 (1787830285) [ 665.637574] LNetError: 32232:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.206.35@tcp is being used as a gateway but routing feature is not turned on [ 665.655720] LNetError: 32232:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 19 previous similar messages [ 665.984774] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg634-server [ 667.838842] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 07:31:29 (1787830289) [ 668.005187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 668.050937] LNet: 33297:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 668.061561] LNetError: 33297:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 668.083567] LNet: Removed LNI 192.168.206.34@tcp [ 668.119681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 668.179539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 668.244406] LNet: Added LNI 192.168.206.34@tcp [8/256/0/180] [ 668.964544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.206.[35-40/2]@tcp [ 668.983979] LNet: 33414:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 668.995858] LNet: 33414:0:(router.c:729:lnet_add_route()) Skipped 4 previous similar messages [ 669.033058] LNetError: 33301:0:(lib-lnet.h:1280:lnet_set_route_aliveness()) route to tcp103 through 192.168.206.37@tcp has gone from up to down [ 669.046373] LNetError: 33301:0:(lib-lnet.h:1280:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 669.065935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.206.[35-40/2]@tcp [ 676.482107] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 07:31:38 (1787830298) [ 676.594632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 676.638175] LNet: 34000:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 676.646409] LNetError: 34000:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 676.671550] LNet: Removed LNI 192.168.206.34@tcp [ 676.721298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 676.861480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 676.911501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 676.944379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 677.035941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 677.096434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 677.155858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 677.223279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 677.281847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 677.313851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 677.400399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 677.496278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 677.541795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 677.579858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 677.652155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 677.675938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 677.723435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 677.797468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 677.840560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 677.918430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 678.013334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 678.064813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 678.151966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 678.279118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 678.323755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 678.385694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 678.484156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 678.517154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 678.579980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 678.654132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 678.694676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 678.766323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 678.848812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 678.874290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 678.944764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 685.089629] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 07:31:47 (1787830307) [ 687.480938] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg634-server [ 689.409137] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 07:31:51 (1787830311) [ 691.425606] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg634-server [ 693.163967] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 07:31:55 (1787830315) [ 693.373986] LNet: 35404:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 693.387231] LNet: 35404:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 11 previous similar messages [ 693.980507] Key type .llcrypt unregistered [ 693.982367] Key type ._llcrypt unregistered [ 700.250308] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 700.416960] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 700.440439] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 700.445576] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 703.060766] Key type ._llcrypt registered [ 703.062453] Key type .llcrypt registered [ 703.231290] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 703.241309] alg: No test for adler32 (adler32-zlib) [ 704.256681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 704.336599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 704.365230] LNet: Added LNI 192.168.206.34@tcp [8/256/0/180] [ 705.135673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 705.175556] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 707.300950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 707.324930] LNet: Removed LNI 10.1.2.3@tcp [ 707.473368] LNet: 36133:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 707.478817] LNetError: 36133:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 707.497768] LNet: Removed LNI 192.168.206.34@tcp [ 708.072422] Key type .llcrypt unregistered [ 708.074123] Key type ._llcrypt unregistered [ 719.288824] Key type ._llcrypt registered [ 719.294084] Key type .llcrypt registered [ 720.890432] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 07:32:22 (1787830342) [ 721.681379] Key type .llcrypt unregistered [ 721.683209] Key type ._llcrypt unregistered [ 727.289789] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 727.460931] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 727.468858] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 727.473866] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 730.026361] Key type ._llcrypt registered [ 730.029026] Key type .llcrypt registered [ 730.283853] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 730.300307] alg: No test for adler32 (adler32-zlib) [ 731.236888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 731.292564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 731.318087] LNet: Added LNI 192.168.206.34@tcp [8/256/0/180] [ 732.021225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 732.050186] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 734.715684] LNet: 37639:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 734.725674] LNetError: 37639:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 734.737180] LNet: Removed LNI 10.1.2.3@tcp [ 734.747919] LNet: Removed LNI 192.168.206.34@tcp [ 735.382417] Key type .llcrypt unregistered [ 735.385192] Key type ._llcrypt unregistered [ 746.567424] Key type ._llcrypt registered [ 746.576473] Key type .llcrypt registered [ 747.986994] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 07:32:49 (1787830369) [ 748.571414] Key type .llcrypt unregistered [ 748.573720] Key type ._llcrypt unregistered [ 754.606598] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 754.722074] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 754.729434] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 754.735252] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 757.204519] Key type ._llcrypt registered [ 757.207756] Key type .llcrypt registered [ 757.376681] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 757.392911] alg: No test for adler32 (adler32-zlib) [ 758.301315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 758.374605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 758.406391] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 760.019122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 760.040770] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 761.716951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 761.732072] LNet: Removed LNI 10.1.2.3@tcp [ 761.746945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 761.776447] LNet: Removed LNI 10.1.2.31@tcp [ 761.807134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 761.838720] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 762.593250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 762.617966] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 763.492367] LNet: 39567:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 763.503894] LNetError: 39567:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg:0 not found [ 763.509922] LNet: Removed LNI 10.1.2.31@tcp [ 763.534411] LNet: Removed LNI 10.1.2.3@tcp [ 764.261495] Key type .llcrypt unregistered [ 764.266664] Key type ._llcrypt unregistered [ 777.474247] Key type ._llcrypt registered [ 777.478683] Key type .llcrypt registered [ 779.113432] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 07:33:20 (1787830400) [ 779.420101] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 779.439535] alg: No test for adler32 (adler32-zlib) [ 780.312872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 780.370940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 780.404851] LNet: Added LNI 192.168.206.34@tcp [8/256/0/180] [ 781.253990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 781.290430] LNet: 40524:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 781.302391] LNetError: 40524:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 781.317562] LNet: Removed LNI 192.168.206.34@tcp [ 781.344924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 781.419144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.206.34@tcp [ 781.452549] LNet: Added LNI 192.168.206.34@tcp [8/256/0/180] [ 782.623918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.206.34@tcp [ 782.666858] LNet: Removed LNI 192.168.206.34@tcp [ 782.751946] LNet: 40660:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 783.430680] Key type .llcrypt unregistered [ 783.432557] Key type ._llcrypt unregistered [ 797.032721] Key type ._llcrypt registered [ 797.033988] Key type .llcrypt registered [ 798.443678] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 07:33:40 (1787830420) [ 800.459769] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg634-server [ 802.673385] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 07:33:44 (1787830424) [ 803.403627] Key type .llcrypt unregistered [ 803.407089] Key type ._llcrypt unregistered [ 810.261001] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 810.430553] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 810.436596] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 810.439971] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 813.140937] Key type ._llcrypt registered [ 813.143360] Key type .llcrypt registered [ 813.448276] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 813.481434] alg: No test for adler32 (adler32-zlib) [ 814.527215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 814.632709] LNet: Added LNI 192.168.206.34@tcp1 [8/256/0/180] [ 815.447433] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 818.070491] LNet: 42505:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 818.074709] LNetError: 42505:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 818.077945] LNet: Removed LNI 192.168.206.34@tcp1 [ 818.089539] LNet: Removed LNI 10.1.2.3@tcp2 [ 818.638750] Key type .llcrypt unregistered [ 818.647680] Key type ._llcrypt unregistered [ 833.276886] Key type ._llcrypt registered [ 833.285826] Key type .llcrypt registered [ 834.941290] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 07:34:16 (1787830456) [ 835.538984] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 835.557723] alg: No test for adler32 (adler32-zlib) [ 836.526684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 836.607642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 836.661646] LNet: Added LNI 192.168.206.34@tcp [8/256/0/180] [ 837.979983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 838.002652] LNet: 43479:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 838.013860] LNetError: 43479:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 838.031661] LNet: Removed LNI 192.168.206.34@tcp [ 838.051746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 838.201749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 838.257047] LNet: Added LNI 192.168.206.34@tcp [8/256/0/180] [ 846.717033] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 07:34:28 (1787830468) [ 847.413577] LNet: 44071:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 847.419962] LNetError: 44071:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 847.432372] LNet: Removed LNI 192.168.206.34@tcp [ 848.218523] Key type .llcrypt unregistered [ 848.224369] Key type ._llcrypt unregistered [ 855.621763] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 855.655695] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 855.659909] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 855.673665] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 858.137074] Key type ._llcrypt registered [ 858.140073] Key type .llcrypt registered [ 858.261464] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 858.273419] alg: No test for adler32 (adler32-zlib) [ 859.212197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 859.296952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 859.324091] LNet: Added LNI 192.168.206.34@tcp [8/256/0/180] [ 860.061249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 860.095427] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 861.976531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 862.001617] LNet: 44817:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 862.009473] LNetError: 44817:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 862.017210] LNet: Removed LNI 10.1.2.3@tcp [ 862.022651] LNet: Removed LNI 192.168.206.34@tcp [ 862.050928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 862.113953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 862.154453] LNet: Added LNI 192.168.206.34@tcp [8/256/0/180] [ 862.161038] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 862.589057] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 862.627544] LNet: 44844:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 862.637968] LNetError: 44844:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 862.648597] LNetError: 44844:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 862.656560] LNet: Removed LNI 10.1.2.3@tcp [ 862.670601] LNet: Removed LNI 192.168.206.34@tcp [ 862.698653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 862.791898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 862.819111] LNet: Added LNI 192.168.206.34@tcp [8/256/0/180] [ 862.824922] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 863.323964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 863.339727] LNet: Added LNI 192.168.206.34@tcp2 [8/256/0/180] [ 864.420511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 864.438938] LNet: 45007:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 864.450987] LNetError: 45007:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 864.462117] LNetError: 45007:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 864.467608] LNet: Removed LNI 10.1.2.3@tcp [ 864.471829] LNet: Removed LNI 192.168.206.34@tcp [ 864.483473] LNet: Removed LNI 192.168.206.34@tcp2 [ 864.508193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 864.594657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 864.620741] LNet: Added LNI 192.168.206.34@tcp [8/256/0/180] [ 864.625007] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 864.631969] LNet: Added LNI 192.168.206.34@tcp2 [8/256/0/180] [ 870.310567] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 07:34:52 (1787830492) [ 870.446112] LNet: 45600:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 870.456223] LNetError: 45600:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 870.467694] LNetError: 45600:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 870.476665] LNet: Removed LNI 10.1.2.3@tcp [ 870.486099] LNet: Removed LNI 192.168.206.34@tcp [ 870.523899] LNet: Removed LNI 192.168.206.34@tcp2 [ 871.283752] Key type .llcrypt unregistered [ 871.287332] Key type ._llcrypt unregistered [ 877.009722] Key type ._llcrypt registered [ 877.016531] Key type .llcrypt registered [ 877.352675] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 877.392569] alg: No test for adler32 (adler32-zlib) [ 878.406992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 878.520227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 878.557789] LNet: Added LNI 192.168.206.34@tcp [8/256/0/180] [ 879.251259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 879.274202] LNet: 46003:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 879.287442] LNetError: 46003:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 879.297839] LNet: Removed LNI 192.168.206.34@tcp [ 879.331760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 879.372984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 879.391856] LNet: Added LNI 192.168.206.34@tcp [8/256/0/180] [ 879.815248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 879.839157] LNet: 46033:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 879.845831] LNetError: 46033:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 879.861563] LNet: Removed LNI 192.168.206.34@tcp [ 879.897049] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 880.008284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 880.040314] LNet: Added LNI 192.168.206.34@tcp [8/256/0/180] [ 880.621311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 880.670987] LNet: Removed LNI 192.168.206.34@tcp [ 880.715195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 880.818433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 880.847303] LNet: Added LNI 192.168.206.34@tcp [8/256/0/180] [ 881.290219] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 881.396458] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 881.398862] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 881.401067] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 883.759583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 883.779336] LNet: 46141:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 883.787836] LNet: 46141:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 883.794788] LNetError: 46141:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 883.805336] LNetError: 46141:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 883.844288] LNet: Removed LNI 192.168.206.34@tcp [ 883.890850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 883.976939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 883.998479] LNet: Added LNI 192.168.206.34@tcp [8/256/0/180] [ 890.838674] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 07:35:12 (1787830512) [ 891.083429] LNet: 46739:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 891.091850] LNetError: 46739:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 891.104597] LNet: Removed LNI 192.168.206.34@tcp [ 891.778913] Key type .llcrypt unregistered [ 891.784481] Key type ._llcrypt unregistered [ 897.760393] Key type ._llcrypt registered [ 897.773280] Key type .llcrypt registered [ 897.984864] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 898.005753] alg: No test for adler32 (adler32-zlib) [ 898.950462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 899.032435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 899.063137] LNet: Added LNI 192.168.206.34@tcp [8/256/0/180] [ 900.023395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 900.045273] LNet: 47141:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 900.054592] LNetError: 47141:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 900.092288] LNet: Removed LNI 192.168.206.34@tcp [ 900.140603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 900.206515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 900.237289] LNet: Added LNI 192.168.206.34@tcp [8/256/0/180] [ 906.348464] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 07:35:28 (1787830528) [ 906.461569] LNet: 47727:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 906.470037] LNetError: 47727:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 906.488989] LNet: Removed LNI 192.168.206.34@tcp [ 907.312938] Key type .llcrypt unregistered [ 907.320098] Key type ._llcrypt unregistered [ 913.477095] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 913.693372] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 913.712854] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 913.725390] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 916.291232] Key type ._llcrypt registered [ 916.293624] Key type .llcrypt registered [ 916.609254] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 916.637638] alg: No test for adler32 (adler32-zlib) [ 917.612829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 917.687942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 917.732702] LNet: Added LNI 192.168.206.34@tcp [8/256/0/180] [ 918.877225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 918.902606] LNet: 48189:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 918.907916] LNetError: 48189:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 918.933959] LNet: Removed LNI 192.168.206.34@tcp [ 918.974470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 919.029179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 919.045978] LNet: Added LNI 192.168.206.34@tcp [8/256/0/180] [ 926.582445] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 07:35:48 (1787830548) [ 926.690734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 926.721553] LNet: 48787:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 926.729810] LNetError: 48787:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 926.756586] LNet: Removed LNI 192.168.206.34@tcp [ 926.780229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 926.895570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 933.047857] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 07:35:54 (1787830554) [ 933.256762] LNet: 49370:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 934.048981] Key type .llcrypt unregistered [ 934.050691] Key type ._llcrypt unregistered [ 940.348298] Key type ._llcrypt registered [ 940.351470] Key type .llcrypt registered [ 940.547475] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 940.573132] alg: No test for adler32 (adler32-zlib) [ 941.477387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 941.570747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 941.599186] LNet: Added LNI 192.168.206.34@tcp [8/256/0/180] [ 942.370376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 942.384299] LNet: 49771:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 942.388790] LNetError: 49771:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 942.403425] LNet: Removed LNI 192.168.206.34@tcp [ 942.433499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 942.526266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 942.556812] LNet: Added LNI 192.168.206.34@tcp [8/256/0/180] [ 948.572920] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 07:36:10 (1787830570) [ 948.630473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 948.644709] LNet: 50357:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 948.649940] LNetError: 50357:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 948.664137] LNet: Removed LNI 192.168.206.34@tcp [ 948.691491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 948.769247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 954.586296] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 07:36:16 (1787830576) [ 955.944640] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 957.666514] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 07:36:19 (1787830579) [ 959.350784] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 960.846433] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 07:36:22 (1787830582) [ 962.501165] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 964.086665] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 07:36:26 (1787830586) [ 965.717284] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 967.433826] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 07:36:29 (1787830589) [ 969.212919] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 970.609766] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing ========================================================== 07:36:32 (1787830592) [ 970.858797] LNet: 52416:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 971.435533] Key type .llcrypt unregistered [ 971.439450] Key type ._llcrypt unregistered [ 976.909644] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 977.083333] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 977.089252] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 977.100506] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 979.510850] Key type ._llcrypt registered [ 979.512846] Key type .llcrypt registered [ 979.643242] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 979.657581] alg: No test for adler32 (adler32-zlib) [ 980.653994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 980.780861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 980.842844] LNet: Added LNI 192.168.206.34@tcp [8/512/0/180] [ 980.857573] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 980.862531] LNet: 52755:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 980.978273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 981.000292] LNet: 52774:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 981.009272] LNetError: 52774:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 981.015953] LNet: Removed LNI 10.1.2.3@tcp [ 981.018735] LNetError: 52774:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 981.047401] LNet: Removed LNI 192.168.206.34@tcp [ 981.077662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 981.150656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 981.192498] LNet: Added LNI 192.168.206.34@tcp [8/256/0/180] [ 981.263869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 981.297063] LNet: 52798:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 981.303228] LNetError: 52798:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 981.321705] LNet: Removed LNI 192.168.206.34@tcp [ 981.346402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 981.422162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 981.445819] LNet: Added LNI 192.168.206.34@tcp [8/256/0/180] [ 981.450288] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 981.456810] LNet: 52806:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 981.548506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 981.569452] LNet: 52824:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 981.577660] LNetError: 52824:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 981.585978] LNet: Removed LNI 10.1.2.3@tcp [ 981.587609] LNetError: 52824:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 981.612656] LNet: Removed LNI 192.168.206.34@tcp [ 981.647359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 981.764025] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 981.808803] LNet: Added LNI 192.168.206.34@tcp [8/256/0/180] [ 981.823674] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 981.830386] LNet: 52832:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 982.067176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 982.098521] LNet: 52858:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 982.104702] LNetError: 52858:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 982.113091] LNet: Removed LNI 10.1.2.3@tcp [ 982.116529] LNetError: 52858:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 982.128783] LNet: Removed LNI 192.168.206.34@tcp [ 982.143673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 982.209297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 982.232700] LNet: Added LNI 192.168.206.34@tcp [8/256/0/182] [ 982.238064] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 982.310355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 982.339322] LNet: 52886:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 982.347232] LNetError: 52886:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 982.359166] LNet: Removed LNI 10.1.2.3@tcp [ 982.360486] LNetError: 52886:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 982.372760] LNet: Removed LNI 192.168.206.34@tcp [ 982.398582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 982.469187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 982.488609] LNet: Added LNI 192.168.206.34@tcp [8/256/0/182] [ 982.492724] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 982.495977] LNet: 52894:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 982.554220] LNet: Removed LNI 10.1.2.3@tcp [ 982.564102] LNet: Removed LNI 192.168.206.34@tcp [ 983.265797] Key type .llcrypt unregistered [ 983.272978] Key type ._llcrypt unregistered [ 994.320890] Key type ._llcrypt registered [ 994.322665] Key type .llcrypt registered [ 995.509420] Lustre: DEBUG MARKER: == sanity-lnet test 180: FastReg map error should not leak descriptors ========================================================== 07:36:57 (1787830617) [ 997.170305] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_180 Need o2ib NETTYPE [ 998.681815] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 07:37:00 (1787830620) [ 999.009540] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 999.024440] alg: No test for adler32 (adler32-zlib) [ 999.963292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1000.020630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1000.037292] LNet: Added LNI 192.168.206.34@tcp [8/256/0/180] [ 1000.655968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.206.34 [ 1000.674914] LNet: 54162:0:(config.c:1738:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1000.680814] LNet: Removed LNI 192.168.206.34@tcp [ 1006.311262] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 07:37:08 (1787830628) [ 1006.560203] LNet: 54732:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1006.563863] LNetError: 54732:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1006.580193] LNet: Removed LNI 192.168.206.34@tcp [ 1007.177866] Key type .llcrypt unregistered [ 1007.180783] Key type ._llcrypt unregistered [ 1013.518437] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1013.773605] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1016.583270] Key type ._llcrypt registered [ 1016.588967] Key type .llcrypt registered [ 1016.850089] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1016.868119] alg: No test for adler32 (adler32-zlib) [ 1018.179534] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1023.895605] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 07:37:26 (1787830646) [ 1024.242707] LNet: 55652:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1024.248979] LNetError: 55652:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1024.267776] LNet: Removed LNI 10.1.2.3@tcp [ 1024.879871] Key type .llcrypt unregistered [ 1024.883353] Key type ._llcrypt unregistered [ 1031.852476] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1032.139253] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1032.146977] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1032.166653] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1035.174826] Key type ._llcrypt registered [ 1035.178863] Key type .llcrypt registered [ 1035.417878] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1035.431542] alg: No test for adler32 (adler32-zlib) [ 1036.416522] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1042.354892] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 07:37:44 (1787830664) [ 1043.737331] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 1045.333566] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 07:37:47 (1787830667) [ 1045.481923] LNet: 56836:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1045.493196] LNetError: 56836:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1045.520156] LNet: Removed LNI 10.1.2.3@tcp [ 1046.173779] Key type .llcrypt unregistered [ 1046.178510] Key type ._llcrypt unregistered [ 1052.387420] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1052.645648] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1052.648645] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1052.659268] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1055.499745] Key type ._llcrypt registered [ 1055.503203] Key type .llcrypt registered [ 1055.749842] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1055.784540] alg: No test for adler32 (adler32-zlib) [ 1056.740504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1056.830575] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1065.529894] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 07:38:07 (1787830687) [ 1066.070929] LNet: 58078:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1066.076919] LNetError: 58078:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1066.097339] LNet: Removed LNI 10.1.2.3@tcp [ 1066.749390] Key type .llcrypt unregistered [ 1066.753368] Key type ._llcrypt unregistered [ 1073.129586] Key type ._llcrypt registered [ 1073.132203] Key type .llcrypt registered [ 1073.327310] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1073.345131] alg: No test for adler32 (adler32-zlib) [ 1074.261449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1074.305344] LNet: Added LNI 192.168.206.34@tcp [8/256/0/180] [ 1082.400608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.134@tcp [ 1083.329823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.34@tcp -d 192.168.206.134@tcp -m GET -e local_interrupt -r 1 [ 1083.388563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.34@tcp -d 192.168.206.34@tcp -m GET -e local_interrupt -r 1 [ 1083.439444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.134@tcp [ 1083.465484] LNet: There was an unexpected network error while writing to 192.168.206.134: rc = -22 [ 1083.555743] LNet: 1 local NIs in recovery (showing 1): 192.168.206.34@tcp [ 1083.688109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1083.722630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1083.760543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1083.800809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1083.845985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1083.911203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1083.942675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1083.978243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1084.005173] LNet: 58521:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1084.031894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1084.102895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1084.148204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1084.195964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1084.233368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1084.277946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1084.312969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1084.884967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.34@tcp -d 192.168.206.134@tcp -m GET -e local_dropped -r 1 [ 1084.935731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.34@tcp -d 192.168.206.34@tcp -m GET -e local_dropped -r 1 [ 1084.982267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.134@tcp [ 1085.004694] LNet: There was an unexpected network error while writing to 192.168.206.134: rc = -22 [ 1085.237852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1085.268723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1085.306456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1085.336944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1085.375557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1085.440057] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1085.487511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1085.521220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1085.554793] LNet: 58630:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1085.579296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1085.620402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1085.677720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1085.706545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1085.751198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1085.793793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1085.838486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1086.206241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.34@tcp -d 192.168.206.134@tcp -m GET -e local_aborted -r 1 [ 1086.249389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.34@tcp -d 192.168.206.34@tcp -m GET -e local_aborted -r 1 [ 1086.304839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.134@tcp [ 1086.338095] LNet: There was an unexpected network error while writing to 192.168.206.134: rc = -22 [ 1086.567980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1086.604827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1086.625253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1086.650119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1086.670395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1086.699601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1086.724746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1086.756669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1086.777972] LNet: 58739:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1086.816651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1086.868873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1086.900743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1086.932681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1086.964650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1086.993750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1087.026490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1087.384306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.34@tcp -d 192.168.206.134@tcp -m GET -e local_no_route -r 1 [ 1087.428526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.34@tcp -d 192.168.206.34@tcp -m GET -e local_no_route -r 1 [ 1087.466571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.134@tcp [ 1087.619296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1087.653461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1087.691192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1087.727684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1087.767324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1087.818151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1087.860880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1087.918675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1087.946138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1087.992454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1088.023956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1088.046115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1088.067547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1088.091731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1088.115272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1088.577709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.34@tcp -d 192.168.206.134@tcp -m GET -e local_timeout -r 1 [ 1088.621089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.34@tcp -d 192.168.206.34@tcp -m GET -e local_timeout -r 1 [ 1088.649462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.134@tcp [ 1088.661250] LNet: There was an unexpected network error while writing to 192.168.206.134: rc = -22 [ 1088.665101] LNet: Skipped 2 previous similar messages [ 1088.828468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1088.875631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1088.914911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1088.944833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1088.973694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1088.996517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1089.043252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1089.068989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1089.085480] LNet: 58958:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1089.089814] LNet: 58958:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1089.102910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1089.150302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1089.192230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1089.232633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1089.268476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1089.319719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1089.381100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1090.044713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.34@tcp -d 192.168.206.134@tcp -m GET -e local_error -r 1 [ 1090.084241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.34@tcp -d 192.168.206.34@tcp -m GET -e local_error -r 1 [ 1090.108562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.134@tcp [ 1090.317050] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1090.343814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1090.371681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1090.399411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1090.424739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1090.463145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1090.510096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1090.557532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1090.594844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1090.620518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1090.646424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1090.677848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1090.726441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1090.771019] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1090.801327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1091.040361] LNet: 59112:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1091.049522] LNetError: 59112:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1091.060146] LNet: Removed LNI 192.168.206.34@tcp [ 1091.575463] Key type .llcrypt unregistered [ 1091.577415] Key type ._llcrypt unregistered [ 1114.292944] Key type ._llcrypt registered [ 1114.296859] Key type .llcrypt registered [ 1115.445449] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 07:38:57 (1787830737) [ 1117.026277] Key type .llcrypt unregistered [ 1117.039090] Key type ._llcrypt unregistered [ 1124.797183] Key type ._llcrypt registered [ 1124.799569] Key type .llcrypt registered [ 1125.217446] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1125.277737] alg: No test for adler32 (adler32-zlib) [ 1126.307949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1126.352896] LNet: Added LNI 192.168.206.34@tcp [8/256/0/180] [ 1134.108349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.134@tcp [ 1145.149596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1145.190782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1145.236215] LNet: Added LNI 192.168.206.34@tcp1 [8/256/0/180] [ 1146.465572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.34@tcp -d 192.168.206.134@tcp -m GET -e local_interrupt -r 1 [ 1146.505132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.34@tcp -d 192.168.206.134@tcp1 -m GET -e local_interrupt -r 1 [ 1146.546851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.34@tcp -d 192.168.206.34@tcp -m GET -e local_interrupt -r 1 [ 1146.595744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.34@tcp -d 192.168.206.34@tcp1 -m GET -e local_interrupt -r 1 [ 1146.651974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.34@tcp1 -d 192.168.206.134@tcp -m GET -e local_interrupt -r 1 [ 1146.706663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.34@tcp1 -d 192.168.206.134@tcp1 -m GET -e local_interrupt -r 1 [ 1146.784901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.34@tcp1 -d 192.168.206.34@tcp -m GET -e local_interrupt -r 1 [ 1146.830568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.34@tcp1 -d 192.168.206.34@tcp1 -m GET -e local_interrupt -r 1 [ 1146.864374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.134@tcp [ 1146.879381] LNet: There was an unexpected network error while writing to 192.168.206.134: rc = -22 [ 1146.884684] LNet: 1 local NIs in recovery (showing 1): 192.168.206.34@tcp [ 1147.174612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1147.210123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1147.249354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1147.287175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1147.318551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1147.368169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1147.402244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1147.443907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1147.473540] LNet: 60586:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1147.498433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1147.544965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1147.580194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1147.613621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1147.658408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1147.700881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1147.745238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1148.266580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.34@tcp -d 192.168.206.134@tcp -m GET -e local_dropped -r 1 [ 1148.299097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.34@tcp -d 192.168.206.134@tcp1 -m GET -e local_dropped -r 1 [ 1148.334213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.34@tcp -d 192.168.206.34@tcp -m GET -e local_dropped -r 1 [ 1148.386521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.34@tcp -d 192.168.206.34@tcp1 -m GET -e local_dropped -r 1 [ 1148.427802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.34@tcp1 -d 192.168.206.134@tcp -m GET -e local_dropped -r 1 [ 1148.479737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.34@tcp1 -d 192.168.206.134@tcp1 -m GET -e local_dropped -r 1 [ 1148.515635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.34@tcp1 -d 192.168.206.34@tcp -m GET -e local_dropped -r 1 [ 1148.586177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.34@tcp1 -d 192.168.206.34@tcp1 -m GET -e local_dropped -r 1 [ 1148.636058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.134@tcp [ 1148.652435] LNet: There was an unexpected network error while writing to 192.168.206.134: rc = -22 [ 1148.660177] LNet: Skipped 2 previous similar messages [ 1148.842630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1148.870558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1148.893619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1148.920564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1148.955811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1148.986239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1149.024605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1149.046643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1149.059628] LNet: 60709:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1149.077195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1149.100850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1149.144415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1149.183658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1149.221585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1149.262471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1149.294468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1149.890468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.34@tcp -d 192.168.206.134@tcp -m GET -e local_aborted -r 1 [ 1149.930922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.34@tcp -d 192.168.206.134@tcp1 -m GET -e local_aborted -r 1 [ 1149.964725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.34@tcp -d 192.168.206.34@tcp -m GET -e local_aborted -r 1 [ 1149.988124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.34@tcp -d 192.168.206.34@tcp1 -m GET -e local_aborted -r 1 [ 1150.033766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.34@tcp1 -d 192.168.206.134@tcp -m GET -e local_aborted -r 1 [ 1150.078213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.34@tcp1 -d 192.168.206.134@tcp1 -m GET -e local_aborted -r 1 [ 1150.138617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.34@tcp1 -d 192.168.206.34@tcp -m GET -e local_aborted -r 1 [ 1150.167820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.34@tcp1 -d 192.168.206.34@tcp1 -m GET -e local_aborted -r 1 [ 1150.189496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.134@tcp [ 1150.203139] LNet: There was an unexpected network error while writing to 192.168.206.134: rc = -22 [ 1150.208502] LNet: Skipped 2 previous similar messages [ 1150.496360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1150.539675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1150.572703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1150.591209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1150.612163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1150.633522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1150.652915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1150.675613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1150.691279] LNet: 60830:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1150.701433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1150.724530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1150.743128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1150.766909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1150.812891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1150.870287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1150.921849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1151.589540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.34@tcp -d 192.168.206.134@tcp -m GET -e local_no_route -r 1 [ 1151.614912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.34@tcp -d 192.168.206.134@tcp1 -m GET -e local_no_route -r 1 [ 1151.655791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.34@tcp -d 192.168.206.34@tcp -m GET -e local_no_route -r 1 [ 1151.677662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.34@tcp -d 192.168.206.34@tcp1 -m GET -e local_no_route -r 1 [ 1151.702702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.34@tcp1 -d 192.168.206.134@tcp -m GET -e local_no_route -r 1 [ 1151.729386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.34@tcp1 -d 192.168.206.134@tcp1 -m GET -e local_no_route -r 1 [ 1151.761434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.34@tcp1 -d 192.168.206.34@tcp -m GET -e local_no_route -r 1 [ 1151.786585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.34@tcp1 -d 192.168.206.34@tcp1 -m GET -e local_no_route -r 1 [ 1151.816802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.134@tcp [ 1152.071240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1152.105453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1152.139456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1152.176837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1152.213939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1152.242690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1152.285114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1152.325529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1152.357510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1152.394086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1152.430119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1152.457104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1152.486756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1152.517773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1152.543060] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1153.001216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.34@tcp -d 192.168.206.134@tcp -m GET -e local_timeout -r 1 [ 1153.029179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.34@tcp -d 192.168.206.134@tcp1 -m GET -e local_timeout -r 1 [ 1153.056472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.34@tcp -d 192.168.206.34@tcp -m GET -e local_timeout -r 1 [ 1153.090771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.34@tcp -d 192.168.206.34@tcp1 -m GET -e local_timeout -r 1 [ 1153.121400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.34@tcp1 -d 192.168.206.134@tcp -m GET -e local_timeout -r 1 [ 1153.154696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.34@tcp1 -d 192.168.206.134@tcp1 -m GET -e local_timeout -r 1 [ 1153.191319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.34@tcp1 -d 192.168.206.34@tcp -m GET -e local_timeout -r 1 [ 1153.222319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.34@tcp1 -d 192.168.206.34@tcp1 -m GET -e local_timeout -r 1 [ 1153.259485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.134@tcp [ 1153.280382] LNet: There was an unexpected network error while writing to 192.168.206.134: rc = -22 [ 1153.284166] LNet: Skipped 5 previous similar messages [ 1153.477938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1153.504973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1153.534954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1153.569909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1153.613923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1153.641756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1153.670504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1153.699704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1153.720569] LNet: 61072:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1153.724118] LNet: 61072:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1153.742902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1153.776623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1153.809065] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1153.843985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1153.885691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1153.917572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1153.943516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1154.333187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.34@tcp -d 192.168.206.134@tcp -m GET -e local_error -r 1 [ 1154.353647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.34@tcp -d 192.168.206.134@tcp1 -m GET -e local_error -r 1 [ 1154.384951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.34@tcp -d 192.168.206.34@tcp -m GET -e local_error -r 1 [ 1154.430820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.34@tcp -d 192.168.206.34@tcp1 -m GET -e local_error -r 1 [ 1154.466137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.34@tcp1 -d 192.168.206.134@tcp -m GET -e local_error -r 1 [ 1154.491812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.34@tcp1 -d 192.168.206.134@tcp1 -m GET -e local_error -r 1 [ 1154.523851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.34@tcp1 -d 192.168.206.34@tcp -m GET -e local_error -r 1 [ 1154.552827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.34@tcp1 -d 192.168.206.34@tcp1 -m GET -e local_error -r 1 [ 1154.586456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.134@tcp [ 1154.799325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1154.825395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1154.851364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1154.876713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1154.902824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1154.926263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1154.953977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1154.983666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1155.013448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1155.048470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1155.080353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1155.113631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1155.145634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1155.176300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1155.207626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1160.409365] LNet: 61273:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1160.439665] LNetError: 61273:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1160.452607] LNet: Removed LNI 192.168.206.34@tcp [ 1160.476987] LNet: Removed LNI 192.168.206.34@tcp1 [ 1160.883752] Key type .llcrypt unregistered [ 1160.887672] Key type ._llcrypt unregistered [ 1176.713591] Key type ._llcrypt registered [ 1176.720475] Key type .llcrypt registered [ 1177.907956] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 07:39:59 (1787830799) [ 1178.752527] Key type .llcrypt unregistered [ 1178.754336] Key type ._llcrypt unregistered [ 1185.772781] Key type ._llcrypt registered [ 1185.780180] Key type .llcrypt registered [ 1186.254674] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1186.274727] alg: No test for adler32 (adler32-zlib) [ 1187.246050] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1187.278425] LNet: Added LNI 192.168.206.34@tcp [8/256/0/180] [ 1196.266752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.134@tcp [ 1197.038559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.34@tcp -d 192.168.206.134@tcp -m GET -e remote_dropped -r 1 [ 1197.067271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.34@tcp -d 192.168.206.34@tcp -m GET -e remote_dropped -r 1 [ 1197.108806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.134@tcp [ 1197.136759] LNet: There was an unexpected network error while writing to 192.168.206.134: rc = -22 [ 1197.363950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1197.388310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1197.431736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1197.473828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1197.519505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1197.552515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1197.609596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1197.679551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1197.742549] LNet: 62527:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1197.794519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1197.842593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1197.881922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1197.926774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1197.958804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1197.992856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1198.060634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1198.638977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.34@tcp -d 192.168.206.134@tcp -m GET -e remote_error -r 1 [ 1198.676396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.34@tcp -d 192.168.206.34@tcp -m GET -e remote_error -r 1 [ 1198.717538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.134@tcp [ 1198.774379] LNet: There was an unexpected network error while writing to 192.168.206.134: rc = -22 [ 1199.084501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1199.110786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1199.134466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1199.191209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1199.239540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1199.280522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1199.318397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1199.375921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1199.403630] LNet: 62636:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1199.441242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1199.509142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1199.570703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1199.622117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1199.675791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1199.738390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1199.802206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1200.406833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.34@tcp -d 192.168.206.134@tcp -m GET -e remote_timeout -r 1 [ 1200.462270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.34@tcp -d 192.168.206.34@tcp -m GET -e remote_timeout -r 1 [ 1200.513316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.134@tcp [ 1200.548983] LNet: There was an unexpected network error while writing to 192.168.206.134: rc = -22 [ 1200.935917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1201.014195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1201.049688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1201.083154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1201.123586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1201.160799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1201.224920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1201.265236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1201.287355] LNet: 62746:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1201.316452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1201.369828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1201.414620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1201.436274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1201.466589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1201.493449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1201.523974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1201.845975] LNet: 62791:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1201.874918] LNetError: 62791:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1201.892904] LNet: Removed LNI 192.168.206.34@tcp [ 1202.365410] Key type .llcrypt unregistered [ 1202.368184] Key type ._llcrypt unregistered [ 1218.632494] Key type ._llcrypt registered [ 1218.635102] Key type .llcrypt registered [ 1219.750599] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 07:40:41 (1787830841) [ 1220.968443] Key type .llcrypt unregistered [ 1220.972423] Key type ._llcrypt unregistered [ 1226.778168] Key type ._llcrypt registered [ 1226.779701] Key type .llcrypt registered [ 1226.892499] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1226.900599] alg: No test for adler32 (adler32-zlib) [ 1227.804359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1227.833887] LNet: Added LNI 192.168.206.34@tcp [8/256/0/180] [ 1236.512620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.134@tcp [ 1244.731626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1244.812886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1244.859073] LNet: Added LNI 192.168.206.34@tcp1 [8/256/0/180] [ 1246.759833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.34@tcp -d 192.168.206.134@tcp -m GET -e remote_dropped -r 1 [ 1246.834959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.34@tcp -d 192.168.206.134@tcp1 -m GET -e remote_dropped -r 1 [ 1246.914110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.34@tcp -d 192.168.206.34@tcp -m GET -e remote_dropped -r 1 [ 1246.993785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.34@tcp -d 192.168.206.34@tcp1 -m GET -e remote_dropped -r 1 [ 1247.056287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.34@tcp1 -d 192.168.206.134@tcp -m GET -e remote_dropped -r 1 [ 1247.097086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.34@tcp1 -d 192.168.206.134@tcp1 -m GET -e remote_dropped -r 1 [ 1247.142229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.34@tcp1 -d 192.168.206.34@tcp -m GET -e remote_dropped -r 1 [ 1247.199853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.34@tcp1 -d 192.168.206.34@tcp1 -m GET -e remote_dropped -r 1 [ 1247.234683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.134@tcp [ 1247.250637] LNet: There was an unexpected network error while writing to 192.168.206.134: rc = -22 [ 1247.256471] LNet: 1 peer NIs in recovery (showing 1): 192.168.206.134@tcp [ 1247.436581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1247.476538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1247.499635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1247.524658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1247.557387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1247.593181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1247.635839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1247.674872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1247.692810] LNet: 64263:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1247.715096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1247.738175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1247.762063] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1247.780631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1247.802410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1247.836568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1247.858376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1247.903050] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1248.248259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.34@tcp -d 192.168.206.134@tcp -m GET -e remote_error -r 1 [ 1248.274611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.34@tcp -d 192.168.206.134@tcp1 -m GET -e remote_error -r 1 [ 1248.300143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.34@tcp -d 192.168.206.34@tcp -m GET -e remote_error -r 1 [ 1248.324515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.34@tcp -d 192.168.206.34@tcp1 -m GET -e remote_error -r 1 [ 1248.350823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.34@tcp1 -d 192.168.206.134@tcp -m GET -e remote_error -r 1 [ 1248.384631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.34@tcp1 -d 192.168.206.134@tcp1 -m GET -e remote_error -r 1 [ 1248.404174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.34@tcp1 -d 192.168.206.34@tcp -m GET -e remote_error -r 1 [ 1248.434029] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.34@tcp1 -d 192.168.206.34@tcp1 -m GET -e remote_error -r 1 [ 1248.464155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.134@tcp [ 1248.487432] LNet: There was an unexpected network error while writing to 192.168.206.134: rc = -22 [ 1248.498354] LNet: Skipped 2 previous similar messages [ 1248.699636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1248.729649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1248.768184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1248.801789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1248.828503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1248.861786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1248.891232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1248.914806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1248.930553] LNet: 64386:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1248.947867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1248.984637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1249.033390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1249.068332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1249.099258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1249.146258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1249.182907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1249.215228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1249.670637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.34@tcp -d 192.168.206.134@tcp -m GET -e remote_timeout -r 1 [ 1249.707142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.34@tcp -d 192.168.206.134@tcp1 -m GET -e remote_timeout -r 1 [ 1249.739913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.34@tcp -d 192.168.206.34@tcp -m GET -e remote_timeout -r 1 [ 1249.776984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.34@tcp -d 192.168.206.34@tcp1 -m GET -e remote_timeout -r 1 [ 1249.806500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.34@tcp1 -d 192.168.206.134@tcp -m GET -e remote_timeout -r 1 [ 1249.846836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.34@tcp1 -d 192.168.206.134@tcp1 -m GET -e remote_timeout -r 1 [ 1249.878916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.34@tcp1 -d 192.168.206.34@tcp -m GET -e remote_timeout -r 1 [ 1249.913623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.34@tcp1 -d 192.168.206.34@tcp1 -m GET -e remote_timeout -r 1 [ 1249.953892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.134@tcp [ 1249.975047] LNet: There was an unexpected network error while writing to 192.168.206.134: rc = -22 [ 1250.145906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1250.168758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1250.189591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1250.244662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1250.276255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1250.314211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1250.362785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1250.411614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1250.440870] LNet: 64509:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1250.465626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1250.505680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1250.546442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1250.576290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1250.603527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1250.639542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1250.666307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1250.694486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1255.230134] LNet: 64591:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1255.238287] LNetError: 64591:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1256.287432] LNet: Removed LNI 192.168.206.34@tcp [ 1256.290731] LNetError: 64591:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1256.307373] LNet: Removed LNI 192.168.206.34@tcp1 [ 1256.703087] Key type .llcrypt unregistered [ 1256.712484] Key type ._llcrypt unregistered [ 1274.194758] Key type ._llcrypt registered [ 1274.198273] Key type .llcrypt registered [ 1275.706694] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 07:41:37 (1787830897) [ 1277.688962] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 1279.394822] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 07:41:41 (1787830901) [ 1280.025822] Key type .llcrypt unregistered [ 1280.028361] Key type ._llcrypt unregistered [ 1286.188265] Key type ._llcrypt registered [ 1286.190875] Key type .llcrypt registered [ 1286.520753] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1286.539191] alg: No test for adler32 (adler32-zlib) [ 1287.508381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1287.540628] LNet: Added LNI 192.168.206.34@tcp [8/256/0/180] [ 1294.492796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.134@tcp [ 1295.255301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.34@tcp -d 192.168.206.134@tcp -m GET -e network_timeout -r 1 [ 1295.303550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.34@tcp -d 192.168.206.34@tcp -m GET -e network_timeout -r 1 [ 1295.340188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.134@tcp [ 1295.367301] LNet: There was an unexpected network error while writing to 192.168.206.134: rc = -22 [ 1295.643878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1295.702746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1295.764764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1295.776265] LNet: 1 local NIs in recovery (showing 1): 192.168.206.34@tcp [ 1295.799544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1295.858951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1295.898768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1295.956457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1295.999366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1296.031694] LNet: 66140:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1296.073979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1296.146513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1296.210736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1296.267837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1296.317809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1296.389723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1296.777932] LNet: 66183:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1296.785992] LNetError: 66183:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1296.814803] LNet: Removed LNI 192.168.206.34@tcp [ 1297.247481] Key type .llcrypt unregistered [ 1297.249618] Key type ._llcrypt unregistered [ 1314.242340] Key type ._llcrypt registered [ 1314.244672] Key type .llcrypt registered [ 1314.405237] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1314.420384] alg: No test for adler32 (adler32-zlib) [ 1315.289180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1315.320468] LNet: Added LNI 192.168.206.34@tcp [8/256/0/180] [ 1322.070782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.134@tcp [ 1330.589800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1330.629574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1330.673320] LNet: Added LNI 192.168.206.34@tcp1 [8/256/0/180] [ 1332.468720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.34@tcp -d 192.168.206.134@tcp -m GET -e network_timeout -r 1 [ 1332.501211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.34@tcp -d 192.168.206.134@tcp1 -m GET -e network_timeout -r 1 [ 1332.528525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.34@tcp -d 192.168.206.34@tcp -m GET -e network_timeout -r 1 [ 1332.560419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.34@tcp -d 192.168.206.34@tcp1 -m GET -e network_timeout -r 1 [ 1332.590339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.34@tcp1 -d 192.168.206.134@tcp -m GET -e network_timeout -r 1 [ 1332.627914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.34@tcp1 -d 192.168.206.134@tcp1 -m GET -e network_timeout -r 1 [ 1332.667677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.34@tcp1 -d 192.168.206.34@tcp -m GET -e network_timeout -r 1 [ 1332.710301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.34@tcp1 -d 192.168.206.34@tcp1 -m GET -e network_timeout -r 1 [ 1332.752593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.134@tcp [ 1332.782450] LNet: There was an unexpected network error while writing to 192.168.206.134: rc = -22 [ 1332.978725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1333.005424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1333.054404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1333.081553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1333.105846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1333.155823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1333.193635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1333.230844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1333.251390] LNet: 67068:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1333.267558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1333.293247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1333.326141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1333.373255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1333.406333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1333.438608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1337.490411] LNet: There was an unexpected network error while writing to 192.168.206.134: rc = -22 [ 1337.873992] LNet: 67145:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1337.883791] LNetError: 67145:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1337.891506] LNet: Removed LNI 192.168.206.34@tcp [ 1337.904471] LNet: Removed LNI 192.168.206.34@tcp1 [ 1338.402855] Key type .llcrypt unregistered [ 1338.408276] Key type ._llcrypt unregistered [ 1353.605540] Key type ._llcrypt registered [ 1353.608396] Key type .llcrypt registered [ 1354.773805] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 07:42:56 (1787830976) [ 1355.423925] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1355.436924] alg: No test for adler32 (adler32-zlib) [ 1356.420830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1356.533546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1356.568227] LNet: Added LNI 192.168.206.34@tcp [8/256/0/180] [ 1357.530050] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1357.575575] LNet: Added LNI 192.168.206.34@tcp1 [8/256/0/180] [ 1358.574361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.34@tcp [ 1358.668785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1358.760431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1358.822811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1358.887745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1358.933569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1358.990825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.206.34@tcp [ 1359.021625] LNetError: 68245:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.206.34@tcp to recovery [ 1359.583335] LNet: 1 local NIs in recovery (showing 1): 192.168.206.34@tcp [ 1359.583355] LNet: There was an unexpected network error while writing to 192.168.206.34: rc = -22 [ 1361.631625] LNet: There was an unexpected network error while writing to 192.168.206.34: rc = -22 [ 1362.522836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1365.727430] LNet: There was an unexpected network error while writing to 192.168.206.34: rc = -22 [ 1366.147919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1366.184614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1366.257728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1366.285304] LNet: 68352:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1366.291402] LNetError: 68352:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1366.295956] LNet: Removed LNI 192.168.206.34@tcp [ 1366.306718] LNet: Removed LNI 192.168.206.34@tcp1 [ 1366.320600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1366.387464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1366.411725] LNet: Added LNI 192.168.206.34@tcp [8/256/0/180] [ 1367.110111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1367.138110] LNet: Added LNI 192.168.206.34@tcp1 [8/256/0/180] [ 1368.082503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.34@tcp [ 1368.133702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1368.181520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1368.219676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1368.245550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1368.276497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1368.310170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1368.372283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.206.34@tcp [ 1368.395723] LNetError: 68625:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.206.34@tcp to recovery [ 1368.418760] LNet: There was an unexpected network error while writing to 192.168.206.34: rc = -22 [ 1370.901862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1374.563077] LNet: There was an unexpected network error while writing to 192.168.206.34: rc = -22 [ 1374.572138] LNet: Skipped 1 previous similar message [ 1378.876882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1378.910618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1378.984347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1386.814171] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 07:43:28 (1787831008) [ 1386.908869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1386.944405] LNet: 69328:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1386.954486] LNetError: 69328:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1386.960165] LNetError: 69328:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1386.970206] LNet: Removed LNI 192.168.206.34@tcp [ 1386.978623] LNet: Removed LNI 192.168.206.34@tcp1 [ 1387.032778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1387.090225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1387.132611] LNet: Added LNI 192.168.206.34@tcp [8/256/0/180] [ 1388.119116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1388.171372] LNet: Added LNI 192.168.206.34@tcp1 [8/256/0/180] [ 1389.244379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.34@tcp [ 1389.364847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1389.389608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1389.416724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1389.445513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1389.524875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1389.565601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.206.34@tcp --health 0 [ 1389.643208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1390.111272] LNet: 1 peer NIs in recovery (showing 1): 192.168.206.34@tcp [ 1390.111759] LNet: There was an unexpected network error while writing to 192.168.206.34: rc = -22 [ 1390.127554] LNet: Skipped 1 previous similar message [ 1405.104658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1405.277075] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1405.317299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1405.342604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.206.34@tcp --health 500 [ 1405.466374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1412.064889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1412.142965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1412.170324] LNet: 69825:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1412.176486] LNetError: 69825:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1412.186479] LNetError: 69825:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1412.201036] LNet: Removed LNI 192.168.206.34@tcp [ 1412.212819] LNet: Removed LNI 192.168.206.34@tcp1 [ 1412.226068] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1412.284418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1412.310584] LNet: Added LNI 192.168.206.34@tcp [8/256/0/180] [ 1412.971938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1412.994309] LNet: Added LNI 192.168.206.34@tcp1 [8/256/0/180] [ 1413.847605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.34@tcp [ 1413.887974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1413.953835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1413.984886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1414.032149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1414.104591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1414.151963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1414.207526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.206.34@tcp --health 0 [ 1415.328131] LNet: There was an unexpected network error while writing to 192.168.206.34: rc = -22 [ 1415.338940] LNet: Skipped 3 previous similar messages [ 1415.541985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1425.862040] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1425.901266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1425.935463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1433.635438] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 07:44:15 (1787831055) [ 1434.115484] LNet: 70872:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1434.129220] LNetError: 70872:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1434.137109] LNetError: 70872:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1435.167153] LNet: Removed LNI 192.168.206.34@tcp [ 1435.186555] LNet: Removed LNI 192.168.206.34@tcp1 [ 1435.847922] Key type .llcrypt unregistered [ 1435.853627] Key type ._llcrypt unregistered [ 1443.014334] Key type ._llcrypt registered [ 1443.016171] Key type .llcrypt registered [ 1443.352668] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1443.378255] alg: No test for adler32 (adler32-zlib) [ 1444.345343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1444.367935] LNet: Added LNI 192.168.206.34@tcp [8/256/0/180] [ 1454.449751] Lustre: DEBUG MARKER: Initial discovery [ 1454.635968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.206.134@tcp [ 1456.899567] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1456.942840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1456.998214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.206.134@tcp [ 1457.028237] LNet: There was an unexpected network error while writing to 192.168.206.134: rc = -22 [ 1457.064284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1457.108057] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.206.134@tcp -r 1 -m GET -l 3 [ 1457.171829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.206.134@tcp -r 1 -m GET -e local_error [ 1457.209192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.206.134@tcp -r 1 -m PUT -l 6 [ 1457.695183] LNet: 1 local NIs in recovery (showing 1): 192.168.206.34@tcp [ 1458.339307] Lustre: DEBUG MARKER: Force oleg634-client.virtnet to discover 192.168.206.134@tcp (in background) [ 1458.553127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.206.134@tcp [ 1459.797525] Lustre: DEBUG MARKER: Force oleg634-server to discover 192.168.206.34@tcp [ 1461.663251] LNet: There was an unexpected network error while writing to 192.168.206.134: rc = -22 [ 1462.355963] Lustre: DEBUG MARKER: Wait for 71639 [ 1463.721349] Lustre: DEBUG MARKER: Finished wait on 71639 [ 1463.872517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay del -a [ 1463.910322] LNet: There was an unexpected network error while writing to 192.168.206.134: rc = -22 [ 1463.923571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1464.240768] LNet: 72093:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1464.248807] LNetError: 72093:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1465.321132] LNet: Removed LNI 192.168.206.34@tcp [ 1465.796446] Key type .llcrypt unregistered [ 1465.801145] Key type ._llcrypt unregistered [ 1482.929218] Key type ._llcrypt registered [ 1482.936746] Key type .llcrypt registered [ 1484.454611] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 07:45:06 (1787831106) [ 1485.061632] Key type .llcrypt unregistered [ 1485.065694] Key type ._llcrypt unregistered [ 1491.584586] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1491.729711] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1491.734596] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1491.762689] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1494.443296] Key type ._llcrypt registered [ 1494.445441] Key type .llcrypt registered [ 1494.697369] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1494.732400] alg: No test for adler32 (adler32-zlib) [ 1495.627185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1495.709205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1495.738601] LNet: Added LNI 192.168.206.34@tcp [8/256/0/180] [ 1496.512845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1496.536458] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1504.726420] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 07:45:26 (1787831126) [ 1504.893936] LNet: 74176:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1504.899945] LNetError: 74176:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1504.905895] LNet: Removed LNI 10.1.2.3@tcp [ 1504.913168] LNet: Removed LNI 192.168.206.34@tcp [ 1505.545420] Key type .llcrypt unregistered [ 1505.547495] Key type ._llcrypt unregistered [ 1512.592673] Key type ._llcrypt registered [ 1512.594430] Key type .llcrypt registered [ 1512.709570] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1512.722179] alg: No test for adler32 (adler32-zlib) [ 1513.636171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1513.702520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1513.762206] LNet: Added LNI 192.168.206.34@tcp [8/256/0/180] [ 1514.528668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1514.548654] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1515.443441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.206.34@tcp [ 1516.511304] LNetError: Unexpected error -22 connecting to 192.168.206.34@tcp at host 192.168.206.34:988 [ 1521.450452] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 07:45:43 (1787831143) [ 1521.665626] LNet: 75312:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1521.674822] LNetError: 75312:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1521.686835] LNet: Removed LNI 10.1.2.3@tcp [ 1521.700723] LNet: Removed LNI 192.168.206.34@tcp [ 1522.233538] Key type .llcrypt unregistered [ 1522.237616] Key type ._llcrypt unregistered [ 1528.016232] Key type ._llcrypt registered [ 1528.022703] Key type .llcrypt registered [ 1528.247314] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1528.260610] alg: No test for adler32 (adler32-zlib) [ 1529.222962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1529.311744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1529.353686] LNet: Added LNI 192.168.206.34@tcp1 [8/256/0/180] [ 1529.960523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1529.987075] LNet: Added LNI 192.168.206.34@tcp2 [8/256/0/180] [ 1530.803065] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.206.34@tcp1 --nid 192.168.206.34@tcp2 [ 1534.663969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.34@tcp1 192.168.206.34@tcp2 [ 1534.681900] LNetError: 76152:0:(lib-move.c:2101:lnet_handle_find_routed_path()) Unable to send message from 192.168.206.34@tcp1 to 192.168.206.34@tcp2 - Route table may be misconfigured [ 1534.702404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.34@tcp2 192.168.206.34@tcp1 [ 1540.554486] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 07:46:02 (1787831162) [ 1540.730231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1540.758074] LNet: 76725:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1540.770512] LNetError: 76725:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1540.780738] LNet: Removed LNI 192.168.206.34@tcp1 [ 1541.805766] LNet: Removed LNI 192.168.206.34@tcp2 [ 1541.841987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1541.909889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1541.929701] LNet: Added LNI 192.168.206.34@tcp [8/256/0/180] [ 1542.577760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1542.601585] LNet: Added LNI 192.168.206.34@tcp1 [8/256/0/180] [ 1543.381273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.34@tcp [ 1543.405461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.34@tcp 192.168.206.34@tcp [ 1543.430556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.34@tcp1 192.168.206.34@tcp1 [ 1543.482509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.206.34@tcp -d 192.168.206.34@tcp -e network_timeout [ 1543.521336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.206.34@tcp -d 192.168.206.34@tcp1 -e network_timeout [ 1543.549545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.206.34@tcp1 -d 192.168.206.34@tcp -e network_timeout [ 1543.580681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.206.34@tcp1 -d 192.168.206.34@tcp1 -e network_timeout [ 1543.624860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.34@tcp [ 1543.639365] LNet: There was an unexpected network error while writing to 192.168.206.34: rc = -22 [ 1543.664294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1543.748285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1543.825184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1543.967153] LNet: 1 local NIs in recovery (showing 1): 192.168.206.34@tcp [ 1550.181253] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 07:46:11 (1787831171) [ 1550.425175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1550.451514] LNet: 77588:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1550.457980] LNetError: 77588:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1550.463044] LNetError: 77588:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1550.492299] LNet: Removed LNI 192.168.206.34@tcp [ 1551.525874] LNet: Removed LNI 192.168.206.34@tcp1 [ 1551.592574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1551.697984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1551.982590] LNet: 77629:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1552.307797] Key type .llcrypt unregistered [ 1552.309815] Key type ._llcrypt unregistered [ 1563.179290] Key type ._llcrypt registered [ 1563.181629] Key type .llcrypt registered [ 1564.113578] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 07:46:26 (1787831186) [ 1564.553960] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1564.564845] alg: No test for adler32 (adler32-zlib) [ 1565.514957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1566.919449] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1568.730164] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 07:46:30 (1787831190) [ 1568.905789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1568.937349] LNet: 78727:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1568.960935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1569.011357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1569.035944] LNet: Added LNI 192.168.206.34@tcp [8/256/0/180] [ 1569.695318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1569.723706] LNet: Added LNI 192.168.206.34@tcp1 [8/256/0/180] [ 1570.452155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.34@tcp [ 1570.496463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.34@tcp1 [ 1570.526762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.34@tcp1 [ 1576.327646] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 07:46:38 (1787831198) [ 1578.114643] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg634-server [ 1579.543770] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 07:46:41 (1787831201) [ 1581.185684] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg634-server [ 1582.698188] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 07:46:44 (1787831204) [ 1584.550430] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg634-server [ 1586.275336] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 07:46:48 (1787831208) [ 1587.928775] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg634-server [ 1589.335866] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 07:46:51 (1787831211) [ 1591.104962] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg634-server [ 1592.668514] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 07:46:54 (1787831214) [ 1594.575728] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg634-server [ 1595.991272] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 07:46:58 (1787831218) [ 1597.567376] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg634-server [ 1599.132272] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 07:47:01 (1787831221) [ 1600.669186] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg634-server [ 1602.212347] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 07:47:04 (1787831224) [ 1603.797949] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg634-server [ 1605.145283] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 07:47:07 (1787831227) [ 1605.297731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1605.321841] LNet: 82604:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1605.327984] LNetError: 82604:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1606.367133] LNet: Removed LNI 192.168.206.34@tcp [ 1606.371663] LNetError: 82604:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1606.388607] LNet: Removed LNI 192.168.206.34@tcp1 [ 1606.411150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1606.451659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1606.466401] LNet: Added LNI 192.168.206.34@tcp [8/256/0/180] [ 1607.028289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1607.107892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.34@tcp [ 1607.272856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1607.304873] LNet: 82751:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1608.359033] LNet: Removed LNI 192.168.206.34@tcp [ 1608.404218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1608.455321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1608.495251] LNet: Added LNI 192.168.206.34@tcp [8/256/0/180] [ 1609.177246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.34@tcp [ 1609.340690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1609.360722] LNet: 82896:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1609.365678] LNetError: 82896:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1609.370731] LNetError: 82896:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1609.384252] LNet: Removed LNI 192.168.206.34@tcp [ 1609.403412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1609.444540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1609.460341] LNet: Added LNI 192.168.206.34@tcp [8/256/0/180] [ 1610.029648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1610.103342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.34@tcp [ 1610.260732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1611.302796] LNet: Removed LNI 192.168.206.34@tcp [ 1611.339300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1611.399849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1611.425236] LNet: Added LNI 192.168.206.34@tcp [8/256/0/180] [ 1612.149518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.34@tcp [ 1612.325493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1612.363762] LNetError: 83189:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1612.371838] LNetError: 83189:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1612.387157] LNet: Removed LNI 192.168.206.34@tcp [ 1612.421966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1612.473171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1612.498136] LNet: Added LNI 192.168.206.34@tcp [8/256/0/180] [ 1613.067173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1613.125585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.34@tcp [ 1613.318263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1614.374273] LNet: Removed LNI 192.168.206.34@tcp [ 1614.400572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1614.468825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1614.492867] LNet: Added LNI 192.168.206.34@tcp [8/256/0/180] [ 1615.279855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.34@tcp [ 1615.507188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1615.565671] LNet: 83481:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1615.577095] LNet: 83481:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 1616.619464] LNet: Removed LNI 192.168.206.34@tcp [ 1616.656353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1616.738784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1616.769574] LNet: Added LNI 192.168.206.34@tcp [8/256/0/180] [ 1617.381722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1617.486597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.34@tcp [ 1617.669520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1617.695907] LNetError: 83628:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1617.700239] LNetError: 83628:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1617.717969] LNet: Removed LNI 192.168.206.34@tcp [ 1617.746559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1617.786808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1617.811614] LNet: Added LNI 192.168.206.34@tcp [8/256/0/180] [ 1618.394174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.34@tcp [ 1618.637670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1619.687952] LNet: Removed LNI 192.168.206.34@tcp [ 1619.724397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1619.800337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1619.822604] LNet: Added LNI 192.168.206.34@tcp [8/256/0/180] [ 1620.422302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1620.485838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.34@tcp [ 1620.659291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1621.735323] LNet: Removed LNI 192.168.206.34@tcp [ 1621.772526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1621.822943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1621.840022] LNet: Added LNI 192.168.206.34@tcp [8/256/0/180] [ 1622.454890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.34@tcp [ 1622.753860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1623.847697] LNet: Removed LNI 192.168.206.34@tcp [ 1623.899550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1623.987184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1624.046294] LNet: Added LNI 192.168.206.34@tcp [8/256/0/180] [ 1624.744707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1624.829579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.34@tcp [ 1625.053423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1625.074642] LNet: 84213:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1625.080519] LNet: 84213:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 1626.150585] LNet: Removed LNI 192.168.206.34@tcp [ 1626.194539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1626.272682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1626.307838] LNet: Added LNI 192.168.206.34@tcp [8/256/0/180] [ 1627.071590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.34@tcp [ 1629.577845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1629.602949] LNetError: 84377:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1629.611014] LNetError: 84377:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 1629.621531] LNet: Removed LNI 192.168.206.34@tcp [ 1629.637460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1629.678691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1629.707520] LNet: Added LNI 192.168.206.34@tcp [8/256/0/180] [ 1630.379233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1630.432870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1630.467553] LNet: Removed LNI 192.168.206.34@tcp [ 1630.491878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1630.569430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1630.592577] LNet: Added LNI 192.168.206.34@tcp [8/256/0/180] [ 1631.496859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1631.589255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1631.627320] LNet: Removed LNI 192.168.206.34@tcp [ 1631.659282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1631.706918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 128 [ 1631.781616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1631.815096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1631.853735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 1000000000000000000000000 [ 1631.928022] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1631.975691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1632.043322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer foo [ 1632.095854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1632.133206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1632.208335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 8 [ 1632.239264] LNet: Added LNI 192.168.206.34@tcp [8/256/0/180] [ 1632.962317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1633.007410] LNet: Removed LNI 192.168.206.34@tcp [ 1633.037817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1633.099043] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1641.634414] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 07:47:43 (1787831263) [ 1641.772792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1641.810257] LNet: 85393:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1641.819578] LNet: 85393:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 7 previous similar messages [ 1641.870796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1641.963865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1641.998945] LNet: Added LNI 192.168.206.34@tcp231 [8/256/0/180] [ 1643.344903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1643.374737] LNet: Removed LNI 192.168.206.34@tcp231 [ 1643.395197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1643.443819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1643.500718] LNet: Added LNI 192.168.206.34@tcp231 [8/256/0/180] [ 1643.505797] LNet: 85544:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1644.283899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1644.321852] LNet: Removed LNI 192.168.206.34@tcp231 [ 1644.335632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1644.379047] LNet: Added LNI 192.168.206.34@tcp231 [8/256/0/180] [ 1645.582736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1645.603644] LNetError: 85688:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1645.607474] LNetError: 85688:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 1645.623368] LNet: Removed LNI 192.168.206.34@tcp231 [ 1645.647444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1645.698190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1645.734858] LNet: Added LNI 192.168.206.34@tcp231 [8/256/0/0] [ 1645.744496] LNet: 85696:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1646.155476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1646.224268] LNet: Removed LNI 192.168.206.34@tcp231 [ 1646.251204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1646.292486] LNet: Added LNI 192.168.206.34@tcp231 [8/256/0/0] [ 1647.588626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1647.640338] LNet: Removed LNI 192.168.206.34@tcp231 [ 1647.683851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1647.769931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1647.809769] LNet: Added LNI 192.168.206.34@tcp231 [8/256/0/60] [ 1647.815186] LNet: 85848:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1648.326866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1648.356411] LNet: Removed LNI 192.168.206.34@tcp231 [ 1648.379609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1648.415374] LNet: Added LNI 192.168.206.34@tcp231 [8/256/0/60] [ 1649.537732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1649.571655] LNet: Removed LNI 192.168.206.34@tcp231 [ 1649.588385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1649.653900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1649.680811] LNet: Added LNI 192.168.206.34@tcp231 [8/256/0/180] [ 1650.110593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1650.144809] LNet: Removed LNI 192.168.206.34@tcp231 [ 1650.164248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1650.204631] LNet: Added LNI 192.168.206.34@tcp231 [8/256/0/180] [ 1657.896788] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 07:47:59 (1787831279) [ 1658.219145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1658.255985] LNet: Removed LNI 192.168.206.34@tcp231 [ 1658.280619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1658.345674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1658.362275] LNet: Added LNI 192.168.206.34@tcp232 [8/256/0/180] [ 1658.970450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1659.003213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1659.027164] LNet: Removed LNI 192.168.206.34@tcp232 [ 1659.039390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1659.060637] LNet: Added LNI 192.168.206.34@tcp232 [8/256/0/180] [ 1659.627143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1659.661692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1659.693663] LNet: Removed LNI 192.168.206.34@tcp232 [ 1659.708856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1659.729721] LNet: Added LNI 192.168.206.34@tcp232 [8/256/0/180] [ 1660.276677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1660.334828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1660.380962] LNet: Removed LNI 192.168.206.34@tcp232 [ 1660.397821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1660.424264] LNet: Added LNI 192.168.206.34@tcp232 [8/256/0/180] [ 1661.012902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1661.043805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1661.097772] LNet: Removed LNI 192.168.206.34@tcp232 [ 1661.114269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1667.022407] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 07:48:09 (1787831289) [ 1668.205215] Key type .llcrypt unregistered [ 1668.209247] Key type ._llcrypt unregistered [ 1674.044604] Key type ._llcrypt registered [ 1674.053634] Key type .llcrypt registered [ 1674.318639] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1674.332357] alg: No test for adler32 (adler32-zlib) [ 1675.233815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1675.258148] LNet: Added LNI 192.168.206.34@tcp [8/256/0/180] [ 1682.035956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.134@tcp [ 1689.335881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1689.379664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1689.403138] LNet: Added LNI 192.168.206.34@tcp1 [8/256/0/180] [ 1690.847982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.34@tcp -d 192.168.206.134@tcp -m GET -e local_interrupt -r 2 [ 1690.875582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.34@tcp -d 192.168.206.134@tcp1 -m GET -e local_interrupt -r 2 [ 1690.902948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.34@tcp -d 192.168.206.34@tcp -m GET -e local_interrupt -r 2 [ 1690.931354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.34@tcp -d 192.168.206.34@tcp1 -m GET -e local_interrupt -r 2 [ 1690.963623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.34@tcp1 -d 192.168.206.134@tcp -m GET -e local_interrupt -r 2 [ 1691.003645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.34@tcp1 -d 192.168.206.134@tcp1 -m GET -e local_interrupt -r 2 [ 1691.031290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.34@tcp1 -d 192.168.206.34@tcp -m GET -e local_interrupt -r 2 [ 1691.064321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.34@tcp1 -d 192.168.206.34@tcp1 -m GET -e local_interrupt -r 2 [ 1691.085274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.34@tcp 192.168.206.134@tcp [ 1691.098123] LNet: There was an unexpected network error while writing to 192.168.206.134: rc = -22 [ 1691.101547] LNet: 1 local NIs in recovery (showing 1): 192.168.206.34@tcp [ 1691.237817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1691.258910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1691.284356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1691.307662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1691.332617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1691.353307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1691.380234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1691.404023] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1691.415976] LNet: 88503:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1691.426739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1691.445518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1691.468586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1691.490246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1691.511150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1691.530769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1691.813533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.34@tcp -d 192.168.206.134@tcp -m GET -e local_dropped -r 2 [ 1691.838084] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.34@tcp -d 192.168.206.134@tcp1 -m GET -e local_dropped -r 2 [ 1691.864983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.34@tcp -d 192.168.206.34@tcp -m GET -e local_dropped -r 2 [ 1691.893353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.34@tcp -d 192.168.206.34@tcp1 -m GET -e local_dropped -r 2 [ 1691.922505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.34@tcp1 -d 192.168.206.134@tcp -m GET -e local_dropped -r 2 [ 1691.951796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.34@tcp1 -d 192.168.206.134@tcp1 -m GET -e local_dropped -r 2 [ 1691.989140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.34@tcp1 -d 192.168.206.34@tcp -m GET -e local_dropped -r 2 [ 1692.025731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.34@tcp1 -d 192.168.206.34@tcp1 -m GET -e local_dropped -r 2 [ 1692.053416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.34@tcp 192.168.206.134@tcp [ 1692.070244] LNet: There was an unexpected network error while writing to 192.168.206.134: rc = -22 [ 1692.203561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1692.226482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1692.250977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1692.274349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1692.295642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1692.316119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1692.336375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1692.355241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1692.364357] LNet: 88623:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1692.374503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1692.391496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1692.407831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1692.426612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1692.447622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1692.465745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1692.824229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.34@tcp -d 192.168.206.134@tcp -m GET -e local_aborted -r 2 [ 1692.864107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.34@tcp -d 192.168.206.134@tcp1 -m GET -e local_aborted -r 2 [ 1692.900755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.34@tcp -d 192.168.206.34@tcp -m GET -e local_aborted -r 2 [ 1692.949646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.34@tcp -d 192.168.206.34@tcp1 -m GET -e local_aborted -r 2 [ 1692.978908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.34@tcp1 -d 192.168.206.134@tcp -m GET -e local_aborted -r 2 [ 1693.001316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.34@tcp1 -d 192.168.206.134@tcp1 -m GET -e local_aborted -r 2 [ 1693.040592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.34@tcp1 -d 192.168.206.34@tcp -m GET -e local_aborted -r 2 [ 1693.071613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.34@tcp1 -d 192.168.206.34@tcp1 -m GET -e local_aborted -r 2 [ 1693.119338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.34@tcp 192.168.206.134@tcp [ 1693.139852] LNet: There was an unexpected network error while writing to 192.168.206.134: rc = -22 [ 1693.146800] LNet: Skipped 1 previous similar message [ 1693.386724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1693.412140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1693.439897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1693.469388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1693.495708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1693.524725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1693.557369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1693.596411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1693.616592] LNet: 88743:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1693.634633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1693.663677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1693.714757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1693.758705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1693.787887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1693.823845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1694.276156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.34@tcp -d 192.168.206.134@tcp -m GET -e local_no_route -r 2 [ 1694.311805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.34@tcp -d 192.168.206.134@tcp1 -m GET -e local_no_route -r 2 [ 1694.351119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.34@tcp -d 192.168.206.34@tcp -m GET -e local_no_route -r 2 [ 1694.389541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.34@tcp -d 192.168.206.34@tcp1 -m GET -e local_no_route -r 2 [ 1694.457594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.34@tcp1 -d 192.168.206.134@tcp -m GET -e local_no_route -r 2 [ 1694.503688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.34@tcp1 -d 192.168.206.134@tcp1 -m GET -e local_no_route -r 2 [ 1694.548113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.34@tcp1 -d 192.168.206.34@tcp -m GET -e local_no_route -r 2 [ 1694.574828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.34@tcp1 -d 192.168.206.34@tcp1 -m GET -e local_no_route -r 2 [ 1694.599797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.34@tcp 192.168.206.134@tcp [ 1694.806144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1694.849104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1694.876627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1694.914092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1694.970680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1695.005983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1695.035665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1695.056378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1695.094465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1695.121673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1695.150422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1695.187571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1695.209404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1695.232318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1695.665147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.34@tcp -d 192.168.206.134@tcp -m GET -e local_timeout -r 2 [ 1695.702034] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.34@tcp -d 192.168.206.134@tcp1 -m GET -e local_timeout -r 2 [ 1695.746176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.34@tcp -d 192.168.206.34@tcp -m GET -e local_timeout -r 2 [ 1695.780529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.34@tcp -d 192.168.206.34@tcp1 -m GET -e local_timeout -r 2 [ 1695.821356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.34@tcp1 -d 192.168.206.134@tcp -m GET -e local_timeout -r 2 [ 1695.856845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.34@tcp1 -d 192.168.206.134@tcp1 -m GET -e local_timeout -r 2 [ 1695.887380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.34@tcp1 -d 192.168.206.34@tcp -m GET -e local_timeout -r 2 [ 1695.918609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.34@tcp1 -d 192.168.206.34@tcp1 -m GET -e local_timeout -r 2 [ 1695.952372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.34@tcp 192.168.206.134@tcp [ 1695.970793] LNet: There was an unexpected network error while writing to 192.168.206.134: rc = -22 [ 1695.977466] LNet: Skipped 1 previous similar message [ 1696.127699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1696.154450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1696.184634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1696.210993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1696.233926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1696.255364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1696.282815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1696.311302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1696.328717] LNet: 88984:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1696.333807] LNet: 88984:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1696.355435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1696.377335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1696.399061] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1696.421483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1696.449145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1696.477566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1696.739529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.34@tcp -d 192.168.206.134@tcp -m GET -e remote_dropped -r 2 [ 1696.761487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.34@tcp -d 192.168.206.134@tcp1 -m GET -e remote_dropped -r 2 [ 1696.782417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.34@tcp -d 192.168.206.34@tcp -m GET -e remote_dropped -r 2 [ 1696.800793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.34@tcp -d 192.168.206.34@tcp1 -m GET -e remote_dropped -r 2 [ 1696.826231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.34@tcp1 -d 192.168.206.134@tcp -m GET -e remote_dropped -r 2 [ 1696.848092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.34@tcp1 -d 192.168.206.134@tcp1 -m GET -e remote_dropped -r 2 [ 1696.867397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.34@tcp1 -d 192.168.206.34@tcp -m GET -e remote_dropped -r 2 [ 1696.888053] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.34@tcp1 -d 192.168.206.34@tcp1 -m GET -e remote_dropped -r 2 [ 1696.913590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.34@tcp 192.168.206.134@tcp [ 1696.924499] LNet: 1 peer NIs in recovery (showing 1): 192.168.206.134@tcp [ 1697.095730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1697.114420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1697.142217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1697.170474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1697.191850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1697.216505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1697.240390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1697.264867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1697.287654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1697.310533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1697.345733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1697.375440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1697.398399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1697.435803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1702.109376] LNet: 89182:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1702.113516] LNetError: 89182:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1702.118028] LNet: Removed LNI 192.168.206.34@tcp [ 1702.130947] LNet: Removed LNI 192.168.206.34@tcp1 [ 1702.531668] Key type .llcrypt unregistered [ 1702.539184] Key type ._llcrypt unregistered [ 1718.769526] Key type ._llcrypt registered [ 1718.771566] Key type .llcrypt registered [ 1720.304578] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 07:49:01 (1787831341) [ 1720.814466] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1720.843715] alg: No test for adler32 (adler32-zlib) [ 1721.722552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1721.796344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1721.850597] LNet: Added LNI 192.168.206.34@tcp [8/256/0/180] [ 1722.884318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1722.938401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.206.34@tcp [ 1722.976691] LNetError: 90160:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.206.34@tcp to recovery [ 1723.113374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.206.34@tcp [ 1723.214945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1730.871849] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 07:49:12 (1787831352) [ 1731.030318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1731.065157] LNet: 90749:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1731.080666] LNetError: 90749:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1731.108710] LNet: Removed LNI 192.168.206.34@tcp [ 1731.130793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1731.195333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1731.226459] LNet: Added LNI 192.168.206.34@tcp [8/256/0/180] [ 1732.063897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1732.080960] LNet: Added LNI 192.168.206.34@tcp1 [8/256/0/180] [ 1733.017991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.206.34@tcp [ 1733.044464] LNetError: 91016:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.206.34@tcp to recovery [ 1733.215654] LNet: 1 local NIs in recovery (showing 1): 192.168.206.34@tcp [ 1733.219481] LNet: There was an unexpected network error while writing to 192.168.206.34: rc = -22 [ 1734.333468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.206.34@tcp [ 1741.479519] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 07:49:23 (1787831363) [ 1743.215849] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg634-server [ 1744.449974] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 07:49:26 (1787831366) [ 1745.940658] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg634-server [ 1747.496224] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 07:49:29 (1787831369) [ 1749.240968] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg634-server [ 1751.003232] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 07:49:32 (1787831372) [ 1751.141688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1751.158739] LNet: 92631:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1751.167393] LNetError: 92631:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1751.178683] LNet: Removed LNI 192.168.206.34@tcp [ 1751.191665] LNet: Removed LNI 192.168.206.34@tcp1 [ 1751.254677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1751.398623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1751.431860] LNet: Added LNI 192.168.206.34@tcp [8/256/0/180] [ 1752.210464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1752.278273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1752.352308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1752.481366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1752.544607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1752.629856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1758.555417] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 07:49:40 (1787831380) [ 1758.743211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1758.755875] LNet: 93365:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1758.764693] LNetError: 93365:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1758.772959] LNetError: 93365:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1758.785972] LNet: Removed LNI 192.168.206.34@tcp [ 1758.814915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1758.883334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1758.932728] LNet: Added LNI 192.168.206.34@tcp [8/256/0/180] [ 1765.739436] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 07:49:47 (1787831387) [ 1765.953936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1765.973963] LNet: 94068:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1765.981794] LNetError: 94068:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1765.995225] LNet: Removed LNI 192.168.206.34@tcp [ 1766.011610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1766.081607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1766.110213] LNet: Added LNI 192.168.206.34@tcp [8/256/0/180] [ 1772.663692] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 07:49:54 (1787831394) [ 1773.989792] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 1775.468375] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 07:49:57 (1787831397) [ 1776.035666] LNet: 95102:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1776.042464] LNetError: 95102:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1776.055892] LNet: Removed LNI 192.168.206.34@tcp [ 1776.708557] Key type .llcrypt unregistered [ 1776.711175] Key type ._llcrypt unregistered [ 1782.154870] Key type ._llcrypt registered [ 1782.158073] Key type .llcrypt registered [ 1782.322891] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1782.338594] alg: No test for adler32 (adler32-zlib) [ 1783.293223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1783.326557] LNet: Added LNI 192.168.206.34@tcp [8/256/0/180] [ 1790.385192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.134@tcp [ 1795.367807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.206.134@tcp [ 1795.844771] LNet: 95527:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1795.851995] LNetError: 95527:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1795.879125] LNet: Removed LNI 192.168.206.34@tcp [ 1796.282487] Key type .llcrypt unregistered [ 1796.285434] Key type ._llcrypt unregistered [ 1807.175096] Key type ._llcrypt registered [ 1807.180055] Key type .llcrypt registered [ 1808.239112] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 1809.409777] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 1810.729771] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 07:50:32 (1787831432) [ 1811.933511] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 [ 1813.284299] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 07:50:35 (1787831435) [ 1814.678330] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg634-server [ 1816.139801] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 07:50:38 (1787831438) [ 1817.896736] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg634-server [ 1819.153637] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 07:50:41 (1787831441) [ 1820.476699] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1820.502951] alg: No test for adler32 (adler32-zlib) [ 1821.584430] Lustre: Lustre: Build Version: 2.17.57_64_gb96373c [ 1821.812029] LNet: Added LNI 192.168.206.34@tcp [8/256/0/180] [ 1823.447782] Key type lgssc registered [ 1824.354391] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1839.768647] Key type lgssc unregistered [ 1840.006523] LNet: 98703:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1840.013385] LNetError: 98703:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1840.024729] LNet: Removed LNI 192.168.206.34@tcp [ 1840.543080] Key type .llcrypt unregistered [ 1840.544856] Key type ._llcrypt unregistered [ 1852.451345] Key type ._llcrypt registered [ 1852.453912] Key type .llcrypt registered [ 1853.646159] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 07:51:15 (1787831475) [ 1853.849604] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1853.858776] alg: No test for adler32 (adler32-zlib) [ 1854.732176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1854.775641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1854.798176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1854.828573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1854.856418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1860.777195] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 07:51:22 (1787831482) [ 1861.006478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1861.026037] LNet: 100119:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1861.042450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1861.109922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1861.162873] LNet: Added LNI 192.168.206.34@tcp [8/256/0/180] [ 1861.998755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1862.017349] LNet: 100263:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1862.023835] LNetError: 100263:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1862.033784] LNet: Removed LNI 192.168.206.34@tcp [ 1862.051563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1862.116692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1862.143744] LNet: Added LNI 192.168.206.34@tcp [8/256/0/180] [ 1862.698628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1862.721833] LNetError: 100291:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1862.747467] LNet: Removed LNI 192.168.206.34@tcp [ 1862.775764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1862.847699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1862.882746] LNet: Added LNI 192.168.206.34@tcp [8/256/0/180] [ 1869.358813] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 07:51:31 (1787831491) [ 1869.696889] LNet: 100881:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1869.701249] LNet: 100881:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1869.714081] LNetError: 100881:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1869.745647] LNet: Removed LNI 192.168.206.34@tcp [ 1870.368810] Key type .llcrypt unregistered [ 1870.371691] Key type ._llcrypt unregistered [ 1875.921170] Key type ._llcrypt registered [ 1875.927133] Key type .llcrypt registered [ 1876.121730] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1876.133213] alg: No test for adler32 (adler32-zlib) [ 1877.078712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1877.145616] LNet: Added LNI 192.168.206.34@tcp [8/256/0/180] [ 1877.149458] LNet: *** cfs_fail_loc=e003, val=100*** [ 1877.722590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.34@tcp [ 1877.734822] LNet: *** cfs_fail_loc=e003, val=100*** [ 1877.739055] LNet: Skipped 2 previous similar messages [ 1877.744406] Ping reply received with mapping: [ 1877.746023] found metadata reply NI[192.168.206.34@tcp] [ 1877.751204] 22 ce a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 1877.753441] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 1877.755555] 5a 5a 5a 5a 5a 5a 5a 5a [ 1877.895412] LNet: 101283:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1877.903331] LNetError: 101283:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1877.915867] LNet: Removed LNI 192.168.206.34@tcp [ 1878.631825] Key type .llcrypt unregistered [ 1878.635885] Key type ._llcrypt unregistered [ 1889.632311] Key type ._llcrypt registered [ 1889.636469] Key type .llcrypt registered [ 1890.785349] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 07:51:52 (1787831512) [ 1891.329361] Key type .llcrypt unregistered [ 1891.333065] Key type ._llcrypt unregistered [ 1897.344351] Key type ._llcrypt registered [ 1897.347798] Key type .llcrypt registered [ 1897.585671] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1897.605350] alg: No test for adler32 (adler32-zlib) [ 1898.810594] LNetError: 102341:0:(api-ni.c:2850:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 1899.227479] Key type .llcrypt unregistered [ 1899.231442] Key type ._llcrypt unregistered [ 1904.134303] Key type ._llcrypt registered [ 1904.136604] Key type .llcrypt registered [ 1905.267564] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 07:52:07 (1787831527) [ 1905.699825] Key type .llcrypt unregistered [ 1905.704133] Key type ._llcrypt unregistered [ 1911.212229] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1911.217470] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1913.727710] Key type ._llcrypt registered [ 1913.731130] Key type .llcrypt registered [ 1913.984868] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1913.998385] alg: No test for adler32 (adler32-zlib) [ 1914.898940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1914.948690] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1923.449891] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 07:52:25 (1787831545) [ 1923.579777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1923.627151] LNet: 104125:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1923.638110] LNetError: 104125:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1923.670043] LNet: Removed LNI 10.1.2.3@tcp [ 1923.694655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1924.493929] Key type .llcrypt unregistered [ 1924.498243] Key type ._llcrypt unregistered [ 1929.603674] Key type ._llcrypt registered [ 1929.607610] Key type .llcrypt registered [ 1929.808395] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1929.819835] alg: No test for adler32 (adler32-zlib) [ 1930.707569] LNet: Added LNI 192.168.206.34@tcp [8/256/0/180] [ 1930.758154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.34@tcp [ 1930.888540] LNet: 104422:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1930.893984] LNetError: 104422:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1931.953418] LNet: Removed LNI 192.168.206.34@tcp [ 1932.558939] Key type .llcrypt unregistered [ 1932.566745] Key type ._llcrypt unregistered [ 1943.717147] Key type ._llcrypt registered [ 1943.719644] Key type .llcrypt registered [ 1945.237427] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 07:52:46 (1787831566) [ 1945.945830] Key type .llcrypt unregistered [ 1945.948825] Key type ._llcrypt unregistered [ 1951.500125] Key type ._llcrypt registered [ 1951.504304] Key type .llcrypt registered [ 1951.730277] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1951.750954] alg: No test for adler32 (adler32-zlib) [ 1959.150538] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 07:53:01 (1787831581) [ 1959.450256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1959.499238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1959.612723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1959.669853] LNet: Added LNI 192.168.206.34@tcp [8/256/0/180] [ 1960.465439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1960.498176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 1960.550256] LNet: Removed LNI 192.168.206.34@tcp [ 1960.801587] LNet: 106178:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1961.108688] Key type .llcrypt unregistered [ 1961.110784] Key type ._llcrypt unregistered [ 1972.107809] Key type ._llcrypt registered [ 1972.109690] Key type .llcrypt registered [ 1973.292238] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 07:53:15 (1787831595) [ 1973.721659] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1973.748508] alg: No test for adler32 (adler32-zlib) [ 1974.626099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1974.668425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1974.696439] LNet: Added LNI 192.168.206.34@tcp [8/256/0/180] [ 1975.421418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.34@tcp [ 1975.454201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.206.34@tcp [ 1975.468361] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 1975.471710] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 1975.473987] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1975, 0, 0 [ 1975.477755] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 1975.479807] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 1975.482762] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 1975.485379] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1975, 0, 0 [ 1975.491964] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 1975.494762] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1975.498465] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 1975.501646] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1975, 0, 2025 [ 1975.505565] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 1975.507663] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1975.510645] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 1975.513517] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1975, 0, 2025 [ 1975.517779] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 1981.692992] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 07:53:23 (1787831603) [ 1982.374208] LNet: 107718:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1982.382452] LNetError: 107718:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1982.393390] LNet: Removed LNI 192.168.206.34@tcp [ 1983.022328] Key type .llcrypt unregistered [ 1983.024040] Key type ._llcrypt unregistered [ 1990.444350] Key type ._llcrypt registered [ 1990.449593] Key type .llcrypt registered [ 1990.661241] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1990.672931] alg: No test for adler32 (adler32-zlib) [ 1991.639570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1991.671496] LNet: Added LNI 192.168.206.34@tcp [8/256/0/180] [ 1999.185221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.134@tcp [ 2009.247418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2009.304642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2009.328908] LNet: Added LNI 192.168.206.34@tcp1 [8/256/0/180] [ 2010.128336] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2010.324105] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2010.328245] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2010.341553] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2012.665985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 2012.686574] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 2022.899912] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 07:54:05 (1787831645) [ 2023.060374] LNet: 109230:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2023.067902] LNetError: 109230:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2023.076941] LNet: Removed LNI 192.168.206.34@tcp [ 2024.095569] LNet: Removed LNI 192.168.206.34@tcp1 [ 2024.101976] LNetError: 109230:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2024.115624] LNetError: 109230:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2024.135649] LNet: Removed LNI 10.1.2.3@tcp99 [ 2024.698603] Key type .llcrypt unregistered [ 2024.702206] Key type ._llcrypt unregistered [ 2030.063053] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2030.069995] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2032.444832] Key type ._llcrypt registered [ 2032.446514] Key type .llcrypt registered [ 2032.583659] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2032.601704] alg: No test for adler32 (adler32-zlib) [ 2033.461377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2033.513623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2033.532630] LNet: Added LNI 192.168.206.34@tcp [8/256/0/180] [ 2034.032481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2034.046108] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2035.633702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 2035.664851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2035.699759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 2035.721483] LNetError: 109982:0:(peer.c:2362:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 2035.747501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2035.788965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.34@tcp [ 2036.050894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2036.113338] LNet: 110002:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2036.117588] LNetError: 110002:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2036.121695] LNet: Removed LNI 10.1.2.3@tcp [ 2037.156918] LNet: Removed LNI 192.168.206.34@tcp [ 2037.710977] Key type .llcrypt unregistered [ 2037.713287] Key type ._llcrypt unregistered [ 2047.947677] Key type ._llcrypt registered [ 2047.949754] Key type .llcrypt registered [ 2048.982727] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 07:54:31 (1787831671) [ 2049.386841] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2049.409169] alg: No test for adler32 (adler32-zlib) [ 2050.404937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2050.445510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2050.463539] LNet: Added LNI 192.168.206.34@tcp [8/256/0/180] [ 2051.108570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.34@tcp [ 2051.188931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg634-client@tcp [ 2056.759978] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 07:54:38 (1787831678) [ 2056.943618] LNet: 111541:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2056.953193] LNetError: 111541:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2056.972279] LNet: Removed LNI 192.168.206.34@tcp [ 2057.479648] Key type .llcrypt unregistered [ 2057.483568] Key type ._llcrypt unregistered [ 2062.901507] Key type ._llcrypt registered [ 2062.903202] Key type .llcrypt registered [ 2063.112807] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2063.132994] alg: No test for adler32 (adler32-zlib) [ 2064.091994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 2064.143821] LNet: Added LNI 192.168.206.34@tcp [8/256/0/180] [ 2064.177887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 2070.061154] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 07:54:52 (1787831692) [ 2071.391616] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 2072.701880] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 07:54:54 (1787831694) [ 2073.952682] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 2075.651302] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 07:54:57 (1787831697) [ 2075.725381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2075.748549] LNet: 112960:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2075.752725] LNetError: 112960:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2075.773561] LNet: Removed LNI 192.168.206.34@tcp [ 2075.797620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2075.844985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2075.864662] LNet: Added LNI 192.168.206.34@tcp [8/256/0/180] [ 2076.489947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2076.530871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2081.912272] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 07:55:04 (1787831704) [ 2082.070814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2082.088967] LNet: 113673:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2082.095652] LNetError: 113673:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2082.107705] LNet: Removed LNI 192.168.206.34@tcp [ 2082.130422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2082.196501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 2082.228097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2082.912672] Key type .llcrypt unregistered [ 2082.917246] Key type ._llcrypt unregistered [ 2091.976272] Key type ._llcrypt registered [ 2091.978898] Key type .llcrypt registered [ 2092.902656] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 07:55:15 (1787831715) [ 2093.322315] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2093.331180] alg: No test for adler32 (adler32-zlib) [ 2094.214726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2094.264882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2094.287949] LNet: Added LNI 192.168.206.34@tcp [8/256/0/180] [ 2094.871599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2094.955583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.34@tcp [ 2100.992662] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 07:55:23 (1787831723) [ 2101.182657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2101.203208] LNet: 115231:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2101.207936] LNetError: 115231:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2101.226335] LNet: Removed LNI 192.168.206.34@tcp [ 2101.241543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2101.276550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 2101.306672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 2107.053975] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 07:55:29 (1787831729) [ 2107.145212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2107.175129] LNet: 115804:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2107.211751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2107.303624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2107.718270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2107.750931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2107.823107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 2108.164500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2108.199598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2108.250844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2108.265860] LNet: 115845:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2108.285438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2108.357989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2108.660528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2108.686864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2108.731655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 2108.985497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2109.008377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2114.306564] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 07:55:36 (1787831736) [ 2114.408957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2114.442342] LNet: 116440:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2114.478235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2114.556115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2114.579939] LNet: Added LNI 192.168.206.34@tcp [8/256/0/180] [ 2115.278978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2115.300753] LNet: Added LNI 192.168.206.34@tcp2 [8/256/0/180] [ 2116.158159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2116.201688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.34@tcp [ 2116.341885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2116.361627] LNetError: 116724:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2117.407200] LNet: Removed LNI 192.168.206.34@tcp [ 2117.411429] LNetError: 116724:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2117.440057] LNet: Removed LNI 192.168.206.34@tcp2 [ 2117.467393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2117.536662] LNet: Added LNI 192.168.206.34@tcp [8/256/0/180] [ 2117.548910] LNet: Added LNI 192.168.206.34@tcp2 [8/256/0/180] [ 2117.556404] LNet: 116731:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2118.217701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.34@tcp [ 2118.459687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2118.476048] LNet: 116768:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2118.488063] LNet: 116768:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 2119.519951] LNet: Removed LNI 192.168.206.34@tcp [ 2119.530179] LNetError: 116768:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2119.544494] LNetError: 116768:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2119.557742] LNet: Removed LNI 192.168.206.34@tcp2 [ 2119.578197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2119.644459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2119.674355] LNet: Added LNI 192.168.206.34@tcp [8/256/0/180] [ 2120.422635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2120.461787] LNet: Added LNI 192.168.206.34@tcp2 [8/256/0/180] [ 2121.340581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.34@tcp [ 2121.375715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2121.434374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.34@tcp [ 2121.653671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2122.719346] LNet: Removed LNI 192.168.206.34@tcp [ 2123.752640] LNet: Removed LNI 192.168.206.34@tcp2 [ 2123.779831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2123.859192] LNet: Added LNI 192.168.206.34@tcp [8/256/0/180] [ 2123.874437] LNet: Added LNI 192.168.206.34@tcp2 [8/256/0/180] [ 2123.888440] LNet: 117060:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2124.416605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.34@tcp [ 2124.617620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2124.638895] LNetError: 117097:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2124.644994] LNetError: 117097:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 2124.652742] LNet: Removed LNI 192.168.206.34@tcp [ 2125.675414] LNet: Removed LNI 192.168.206.34@tcp2 [ 2125.718221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2125.791665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2125.826544] LNet: Added LNI 192.168.206.34@tcp [8/256/0/180] [ 2126.475883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2126.491439] LNet: Added LNI 192.168.206.34@tcp2 [8/256/0/180] [ 2127.320655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2127.374264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.34@tcp [ 2127.580772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2127.601590] LNet: 117376:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2127.605960] LNet: 117376:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 2128.671238] LNet: Removed LNI 192.168.206.34@tcp [ 2128.682391] LNet: Removed LNI 192.168.206.34@tcp2 [ 2128.710467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2128.765626] LNet: Added LNI 192.168.206.34@tcp [8/256/0/180] [ 2128.773175] LNet: Added LNI 192.168.206.34@tcp2 [8/256/0/180] [ 2128.778272] LNet: 117382:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2129.335202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.34@tcp [ 2129.536190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2130.591206] LNet: Removed LNI 192.168.206.34@tcp [ 2130.602995] LNet: Removed LNI 192.168.206.34@tcp2 [ 2130.637926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2130.687263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2130.708607] LNet: Added LNI 192.168.206.34@tcp [8/256/0/180] [ 2131.323327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2131.338341] LNet: Added LNI 192.168.206.34@tcp2 [8/256/0/180] [ 2132.047172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.34@tcp [ 2132.079841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2132.122767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.34@tcp [ 2132.385026] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2133.471164] LNet: Removed LNI 192.168.206.34@tcp [ 2133.474415] LNetError: 117704:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2133.480225] LNetError: 117704:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 6 previous similar messages [ 2133.499815] LNet: Removed LNI 192.168.206.34@tcp2 [ 2133.536482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2133.625169] LNet: Added LNI 192.168.206.34@tcp [8/256/0/180] [ 2133.638945] LNet: Added LNI 192.168.206.34@tcp2 [8/256/0/180] [ 2133.642768] LNet: 117710:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2134.143405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.34@tcp [ 2134.371290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2134.387962] LNet: Removed LNI 192.168.206.34@tcp [ 2135.402492] LNet: Removed LNI 192.168.206.34@tcp2 [ 2135.478426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2135.541160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2135.580473] LNet: Added LNI 192.168.206.34@tcp [8/256/0/180] [ 2136.222785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2136.247377] LNet: Added LNI 192.168.206.34@tcp2 [8/256/0/180] [ 2137.219932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2137.250897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2137.304378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.34@tcp [ 2137.586710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2138.655220] LNet: Removed LNI 192.168.206.34@tcp [ 2138.665413] LNet: Removed LNI 192.168.206.34@tcp2 [ 2138.722487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2138.812063] LNet: Added LNI 192.168.206.34@tcp [8/256/0/180] [ 2138.822305] LNet: Added LNI 192.168.206.34@tcp2 [8/256/0/180] [ 2138.826551] LNet: 118050:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2139.370873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.34@tcp [ 2139.849663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2140.895143] LNet: Removed LNI 192.168.206.34@tcp [ 2140.904614] LNet: Removed LNI 192.168.206.34@tcp2 [ 2140.949411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2141.013862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2141.051764] LNet: Added LNI 192.168.206.34@tcp [8/256/0/180] [ 2141.753827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2141.790533] LNet: Added LNI 192.168.206.34@tcp2 [8/256/0/180] [ 2142.728353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.34@tcp [ 2142.783579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2142.837628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2142.905597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.34@tcp [ 2143.270861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2144.351159] LNet: Removed LNI 192.168.206.34@tcp [ 2145.386295] LNet: Removed LNI 192.168.206.34@tcp2 [ 2145.410395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2145.464869] LNet: Added LNI 192.168.206.34@tcp [8/256/0/180] [ 2145.469170] LNet: Added LNI 192.168.206.34@tcp2 [8/256/0/180] [ 2145.985391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.34@tcp [ 2152.410519] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 2153.935927] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 07:56:15 (1787831775) [ 2154.041470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2154.085646] LNet: 119164:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2154.090061] LNet: 119164:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 6 previous similar messages [ 2154.095989] LNetError: 119164:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2154.101383] LNetError: 119164:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [ 2154.108587] LNet: Removed LNI 192.168.206.34@tcp [ 2155.178101] LNet: Removed LNI 192.168.206.34@tcp2 [ 2155.230628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2155.550751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2155.578533] LNet: Added LNI 192.168.206.34@tcp [8/256/0/180] [ 2156.404360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2156.437718] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2157.293410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2157.328604] LNet: Removed LNI 10.1.2.3@tcp [ 2157.344428] LNet: Removed LNI 192.168.206.34@tcp [ 2157.370602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2157.447237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 2157.475362] LNet: Added LNI 192.168.206.34@tcp [8/256/0/180] [ 2157.491029] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2157.963669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 2157.980980] LNet: Removed LNI 192.168.206.34@tcp [ 2157.993960] LNet: Removed LNI 10.1.2.3@tcp [ 2164.439580] Lustre: DEBUG MARKER: == sanity-lnet test 475: per-operation latency stats populate and reset ========================================================== 07:56:26 (1787831786) [ 2164.617784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2164.646526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2164.711558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2164.742105] LNet: Added LNI 192.168.206.34@tcp [8/256/0/180] [ 2165.423534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2165.458650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.34@tcp [ 2165.487944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.34@tcp [ 2165.528695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.34@tcp [ 2165.571359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.34@tcp [ 2165.597568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.34@tcp [ 2165.620196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.34@tcp [ 2165.642980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.34@tcp [ 2165.669399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.34@tcp [ 2165.724865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.34@tcp [ 2165.755269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.34@tcp [ 2165.901832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2171.852630] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 07:56:33 (1787831793) [ 2172.082815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2173.165935] LNet: Removed LNI 192.168.206.34@tcp [ 2173.210605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2173.345549] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2173.415317] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2173.418538] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2173.423787] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2175.833309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2175.858823] LNet: Added LNI 192.168.206.34@tcp [8/256/0/180] [ 2176.554802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2176.585638] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2178.130138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.34@tcp [ 2178.160460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 2178.211466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.206.34@tcp -m PUT -e local_timeout -r 1 [ 2178.296265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 2178.363387] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2178.367217] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2178.628227] LNetError: 120851:0:(lib-move.c:4801:lnet_parse()) 192.168.206.34@tcp, src 192.168.206.34@tcp: Dropping PUT (error -108 looking up sender) [ 2179.680208] LNet: Removed LNI 10.1.2.3@tcp [ 2180.717590] LNet: Removed LNI 192.168.206.34@tcp [ 2180.973488] Key type .llcrypt unregistered [ 2180.976917] Key type ._llcrypt unregistered [ 2192.930981] Key type ._llcrypt registered [ 2192.933364] Key type .llcrypt registered [ 2194.246134] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 07:56:55 (1787831815) [ 2194.629405] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2194.645534] alg: No test for adler32 (adler32-zlib) [ 2195.581149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2195.947741] LNet: 122165:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2196.570790] Key type .llcrypt unregistered [ 2196.572555] Key type ._llcrypt unregistered [ 2202.518688] Key type ._llcrypt registered [ 2202.521058] Key type .llcrypt registered [ 2202.714607] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2202.730605] alg: No test for adler32 (adler32-zlib) [ 2203.642818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2203.667442] LNet: Added LNI 192.168.206.34@tcp [8/256/0/180] [ 2211.053254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.134@tcp [ 2211.440361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 2212.221872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.206.134@tcp [ 2212.524819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2212.562396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2212.626454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2212.666837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2212.709662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2212.737059] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2212.774468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2212.809083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2212.842127] LNet: 122618:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2212.864179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2212.895154] LNet: 1 peer NIs in recovery (showing 1): 192.168.206.134@tcp [ 2212.909178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2212.962850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2213.013968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2213.052830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2213.100983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2213.748332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2213.963260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2213.989844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2214.016275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2214.041547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2214.076480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2214.103866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2214.134784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2214.163582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2214.173737] LNet: 122736:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2214.188601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2214.218683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2214.237478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2214.270500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2214.297667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2214.329234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2214.845713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.206.134@tcp [ 2215.034306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2215.075290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2215.112726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2215.150573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2215.198869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2215.236173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2215.263531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2215.297874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2215.314783] LNet: 122853:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2215.350017] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2215.409293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2215.444248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2215.484425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2215.523353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2215.555452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2216.154959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2216.393904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2216.417143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2216.451942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2216.503700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2216.538141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2216.567846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2216.601858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2216.650768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2216.689936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2216.743804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2216.798888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2216.845385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2216.877702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2216.903620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2217.353234] LNet: 123020:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2217.359877] LNetError: 123020:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2217.378326] LNet: Removed LNI 192.168.206.34@tcp [ 2217.762853] Key type .llcrypt unregistered [ 2217.768584] Key type ._llcrypt unregistered [ 2236.907164] Key type ._llcrypt registered [ 2236.911396] Key type .llcrypt registered [ 2238.196563] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 07:57:40 (1787831860) [ 2238.681185] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2238.721949] alg: No test for adler32 (adler32-zlib) [ 2239.705351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2240.111994] LNet: 123908:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2240.699785] Key type .llcrypt unregistered [ 2240.705432] Key type ._llcrypt unregistered [ 2247.314207] Key type ._llcrypt registered [ 2247.315872] Key type .llcrypt registered [ 2247.586701] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2247.615548] alg: No test for adler32 (adler32-zlib) [ 2248.609802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2248.645201] LNet: Added LNI 192.168.206.34@tcp [8/256/0/180] [ 2256.077094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.134@tcp [ 2265.028604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2265.106743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2265.139280] LNet: Added LNI 192.168.206.34@tcp1 [8/256/0/180] [ 2266.023702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 2266.652280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.206.134@tcp [ 2266.865182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2266.892584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2266.920126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2266.949106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2266.975499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2267.002568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2267.031200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2267.058319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2267.084437] LNet: 124564:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2267.105416] LNet: 1 peer NIs in recovery (showing 1): 192.168.206.134@tcp [ 2267.111751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2267.155712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2267.192392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2267.229543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2267.279722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2267.340942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2267.990944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2268.194788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2268.217958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2268.262658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2268.319959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2268.363973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2268.391780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2268.412436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2268.431562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2268.463533] LNet: 124681:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2268.484658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2268.517171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2268.562510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2268.603716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2268.652166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2268.715053] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2269.324630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.206.134@tcp [ 2269.521660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2269.548549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2269.572718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2269.596723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2269.617733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2269.639418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2269.664508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2269.689608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2269.703204] LNet: 124798:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2269.723829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2269.751437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2269.797898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2269.845478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2269.888464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2269.933345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2270.497828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2270.681622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2270.704185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2270.728523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2270.754480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2270.779105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2270.806895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2270.854278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2270.887678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2270.936500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2270.972697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2271.004469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2271.051264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2271.099377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2271.153776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2276.270955] LNet: 125000:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2276.279167] LNetError: 125000:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2276.284746] LNet: Removed LNI 192.168.206.34@tcp [ 2276.309181] LNet: Removed LNI 192.168.206.34@tcp1 [ 2276.733934] Key type .llcrypt unregistered [ 2276.737493] Key type ._llcrypt unregistered [ 2296.374134] Key type ._llcrypt registered [ 2296.376532] Key type .llcrypt registered [ 2298.234109] Lustre: DEBUG MARKER: == sanity-lnet test 525: keep reachable peer on local net delete ========================================================== 07:58:39 (1787831919) [ 2298.603340] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2298.620172] alg: No test for adler32 (adler32-zlib) [ 2299.514729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2299.562734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2299.584314] LNet: Added LNI 192.168.206.34@tcp [8/256/0/180] [ 2300.305989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2300.339564] LNet: Added LNI 192.168.206.34@tcp2 [8/256/0/180] [ 2301.104527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.34@tcp [ 2301.192535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp [ 2302.240068] LNet: Removed LNI 192.168.206.34@tcp [ 2309.473343] Lustre: DEBUG MARKER: == sanity-lnet test 550: DD-off: keep route up when gw reachable via another local NI ========================================================== 07:58:51 (1787831931) [ 2310.878838] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_550 Need at least 2 local interfaces for multi-rail [ 2312.330806] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 07:58:54 (1787831934) [ 2318.675966] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 07:59:00 (1787831940) [ 2324.885187] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 07:59:07 (1787831947) [ 2330.881901] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 07:59:12 (1787831952) [ 2335.339648] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 07:59:17 (1787831957) [ 2341.514093] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 07:59:23 (1787831963) [ 2348.137412] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 07:59:30 (1787831970) [ 2354.779705] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 07:59:36 (1787831976) [ 2360.365442] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 07:59:42 (1787831982) [ 2367.028447] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 07:59:49 (1787831989) [ 2373.260589] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 07:59:55 (1787831995) [ 2379.457913] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 08:00:01 (1787832001) [ 2386.335510] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 08:00:08 (1787832008) [ 2393.102188] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 08:00:15 (1787832015) [ 2399.485845] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 08:00:21 (1787832021) [ 2405.679339] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 08:00:27 (1787832027) [ 2412.186610] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 08:00:34 (1787832034) [ 2418.206553] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 08:00:40 (1787832040) [ 2418.443794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2418.492360] LNet: 136692:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2418.507329] LNetError: 136692:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2418.536974] LNet: Removed LNI 192.168.206.34@tcp2 [ 2418.583417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2418.743734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2418.785920] LNet: Added LNI 192.168.206.34@tcp [8/256/0/180] [ 2419.987338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2420.003601] LNet: 136834:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2420.009594] LNetError: 136834:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2420.019884] LNet: Removed LNI 192.168.206.34@tcp [ 2420.040697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2420.109683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2420.134240] LNet: Added LNI 192.168.206.34@tcp [8/256/0/180] [ 2426.555786] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 08:00:48 (1787832048) [ 2432.851672] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 08:00:54 (1787832054) [ 2438.709640] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 08:01:00 (1787832060) [ 2445.001318] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 08:01:07 (1787832067) [ 2451.165969] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 08:01:13 (1787832073) [ 2457.406095] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 08:01:19 (1787832079) [ 2463.049950] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 08:01:25 (1787832085) [ 2468.909581] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 08:01:31 (1787832091) [ 2475.024897] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 08:01:37 (1787832097) [ 2481.951224] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 08:01:43 (1787832103) [ 2488.988716] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 08:01:50 (1787832110) [ 2495.763551] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 08:01:57 (1787832117) [ 2502.227784] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 2234 sec ========== 08:02:04 (1787832124) [ 2502.437318] LNet: 144311:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2502.441954] LNetError: 144311:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2502.463618] LNet: Removed LNI 192.168.206.34@tcp [ 2503.028794] Key type .llcrypt unregistered [ 2503.036065] Key type ._llcrypt unregistered [ 2514.354989] Key type ._llcrypt registered [ 2514.364179] Key type .llcrypt registered [ 2514.706840] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2514.728815] alg: No test for adler32 (adler32-zlib) [ 2515.844413] Lustre: Lustre: Build Version: 2.17.57_64_gb96373c [ 2516.137747] LNet: Added LNI 192.168.206.34@tcp [8/256/0/180] [ 2517.807384] Key type lgssc registered [ 2518.819445] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2579.225955] Lustre: Mounted lustre-client - version 2.17.57_64_gb96373c [ 2584.514047] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2592.100884] Lustre: Unmounted lustre-client [ 2616.376478] Key type lgssc unregistered [ 2616.612736] LNet: 147266:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2616.636735] LNetError: 147266:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2616.658178] LNet: Removed LNI 192.168.206.34@tcp [ 2617.282162] Key type .llcrypt unregistered [ 2617.285266] Key type ._llcrypt unregistered