[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 460458366 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.003178] x2apic enabled [ 0.004008] Switched APIC routing to physical x2apic. [ 0.005012] kvm-guest: setup PV IPIs [ 0.008564] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009022] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010012] pid_max: default: 32768 minimum: 301 [ 0.011132] LSM: Security Framework initializing [ 0.012054] Yama: becoming mindful. [ 0.013037] SELinux: Initializing. [ 0.014072] *** VALIDATE selinux *** [ 0.022343] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027087] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028159] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029127] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030115] *** VALIDATE tmpfs *** [ 0.031447] *** VALIDATE proc *** [ 0.033172] *** VALIDATE cgroup *** [ 0.034009] *** VALIDATE cgroup2 *** [ 0.036118] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.037143] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.038008] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.040026] Spectre V2 : User space: Vulnerable [ 0.041009] Speculative Store Bypass: Vulnerable [ 0.044028] debug: unmapping init [mem 0xffffffffb5459000-0xffffffffb5460fff] [ 0.046179] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.047645] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.048026] ... version: 2 [ 0.049014] ... bit width: 48 [ 0.050011] ... generic registers: 4 [ 0.051011] ... value mask: 0000ffffffffffff [ 0.052013] ... max period: 00007fffffffffff [ 0.053014] ... fixed-purpose events: 3 [ 0.054010] ... event mask: 000000070000000f [ 0.055275] rcu: Hierarchical SRCU implementation. [ 0.057415] smp: Bringing up secondary CPUs ... [ 0.058556] x86: Booting SMP configuration: [ 0.059021] .... node #0, CPUs: #1 #2 #3 [ 0.063071] smp: Brought up 1 node, 4 CPUs [ 0.065016] smpboot: Max logical packages: 1 [ 0.066013] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.121483] node 0 deferred pages initialised in 53ms [ 0.125136] devtmpfs: initialized [ 0.126217] x86/mm: Memory block size: 128MB [ 0.128742] gcov: version magic: 0x41383552 [ 0.130169] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.131080] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.132244] pinctrl core: initialized pinctrl subsystem [ 0.133183] [ 0.133811] ************************************************************* [ 0.134014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.135011] ** ** [ 0.136014] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.137014] ** ** [ 0.138012] ** This means that this kernel is built to expose internal ** [ 0.139012] ** IOMMU data structures, which may compromise security on ** [ 0.140010] ** your system. ** [ 0.141011] ** ** [ 0.142011] ** If you see this message and you are not debugging the ** [ 0.143009] ** kernel, report this immediately to your vendor! ** [ 0.144011] ** ** [ 0.145011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.146009] ************************************************************* [ 0.147641] NET: Registered protocol family 16 [ 0.148435] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.149052] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.150049] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.152013] cpuidle: using governor menu [ 0.154131] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.157460] PCI: Using configuration type 1 for base access [ 0.159145] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.168104] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.169033] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.171151] cryptd: max_cpu_qlen set to 1000 [ 0.173055] ACPI: Added _OSI(Module Device) [ 0.174009] ACPI: Added _OSI(Processor Device) [ 0.175008] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.176010] ACPI: Added _OSI(Processor Aggregator Device) [ 0.180560] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.183507] ACPI: Interpreter enabled [ 0.184056] ACPI: PM: (supports S0 S3 S4 S5) [ 0.185011] ACPI: Using IOAPIC for interrupt routing [ 0.186158] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.187436] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.196899] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.197038] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.198018] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.199074] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.201383] acpiphp: Slot [2] registered [ 0.202114] acpiphp: Slot [5] registered [ 0.203091] acpiphp: Slot [6] registered [ 0.204101] acpiphp: Slot [3] registered [ 0.205070] acpiphp: Slot [4] registered [ 0.206075] acpiphp: Slot [7] registered [ 0.207097] acpiphp: Slot [8] registered [ 0.208095] acpiphp: Slot [9] registered [ 0.209076] acpiphp: Slot [10] registered [ 0.210076] acpiphp: Slot [11] registered [ 0.211078] acpiphp: Slot [12] registered [ 0.212093] acpiphp: Slot [13] registered [ 0.213078] acpiphp: Slot [14] registered [ 0.214073] acpiphp: Slot [15] registered [ 0.215159] acpiphp: Slot [16] registered [ 0.216074] acpiphp: Slot [17] registered [ 0.217085] acpiphp: Slot [18] registered [ 0.218073] acpiphp: Slot [19] registered [ 0.219077] acpiphp: Slot [20] registered [ 0.220077] acpiphp: Slot [21] registered [ 0.221112] acpiphp: Slot [22] registered [ 0.222078] acpiphp: Slot [23] registered [ 0.223082] acpiphp: Slot [24] registered [ 0.224100] acpiphp: Slot [25] registered [ 0.225103] acpiphp: Slot [26] registered [ 0.226085] acpiphp: Slot [27] registered [ 0.227083] acpiphp: Slot [28] registered [ 0.228110] acpiphp: Slot [29] registered [ 0.229112] acpiphp: Slot [30] registered [ 0.230087] acpiphp: Slot [31] registered [ 0.231050] PCI host bridge to bus 0000:00 [ 0.233018] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.235036] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.237017] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.239020] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.241093] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.244020] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.245230] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.247021] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.250349] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.257013] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.261051] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.263019] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.264013] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.265014] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.267578] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.269808] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.272036] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.274723] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.277792] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.286015] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.290012] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.297505] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.306062] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.313020] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.331020] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.349860] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.356013] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.366014] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.380010] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.392336] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.395437] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.398379] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.401359] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.404212] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.410026] iommu: Default domain type: Passthrough [ 0.411410] SCSI subsystem initialized [ 0.412120] ACPI: bus type USB registered [ 0.414076] usbcore: registered new interface driver usbfs [ 0.416068] usbcore: registered new interface driver hub [ 0.418114] usbcore: registered new device driver usb [ 0.420222] pps_core: LinuxPPS API ver. 1 registered [ 0.423013] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.425052] PTP clock support registered [ 0.428040] EDAC MC: Ver: 3.0.0 [ 0.429340] PCI: Using ACPI for IRQ routing [ 0.431687] NetLabel: Initializing [ 0.432022] NetLabel: domain hash size = 128 [ 0.434009] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.435103] NetLabel: unlabeled traffic allowed by default [ 0.438114] vgaarb: loaded [ 0.439269] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.442014] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.451859] clocksource: Switched to clocksource kvm-clock [ 0.560317] VFS: Disk quotas dquot_6.6.0 [ 0.565676] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.568237] *** VALIDATE ramfs *** [ 0.569653] *** VALIDATE hugetlbfs *** [ 0.571501] pnp: PnP ACPI init [ 0.574743] pnp: PnP ACPI: found 6 devices [ 0.593284] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.599448] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.601855] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.604217] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.606813] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.609278] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.612576] NET: Registered protocol family 2 [ 0.615469] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.622065] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.626660] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.632515] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.635987] TCP: Hash tables configured (established 65536 bind 65536) [ 0.639246] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.642983] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.646031] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.649548] NET: Registered protocol family 1 [ 0.652070] RPC: Registered named UNIX socket transport module. [ 0.654086] RPC: Registered udp transport module. [ 0.656046] RPC: Registered tcp transport module. [ 0.657629] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.659911] NET: Registered protocol family 44 [ 0.661161] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.663466] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.666451] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.668342] PCI: CLS 0 bytes, default 64 [ 0.669868] Unpacking initramfs... [ 2.071744] debug: unmapping init [mem 0xffff88b43cc64000-0xffff88b43ffcffff] [ 2.075983] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.078544] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.081226] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.595511] Initialise system trusted keyrings [ 2.597191] Key type blacklist registered [ 2.599145] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.608611] zbud: loaded [ 2.611669] *** VALIDATE nfs *** [ 2.612936] *** VALIDATE nfs4 *** [ 2.615228] pstore: using deflate compression [ 2.619965] Platform Keyring initialized [ 2.734770] NET: Registered protocol family 38 [ 2.736400] Key type asymmetric registered [ 2.737756] Asymmetric key parser 'x509' registered [ 2.739900] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.742671] io scheduler mq-deadline registered [ 2.743925] io scheduler kyber registered [ 2.745207] io scheduler bfq registered [ 2.747066] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.749546] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.751993] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.754559] ACPI: Power Button [PWRF] [ 2.759827] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.766290] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.776073] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.803470] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.832737] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.838297] Non-volatile memory driver v1.3 [ 2.840752] Linux agpgart interface v0.103 [ 2.876985] virtio_blk virtio1: [vda] 149768 512-byte logical blocks (76.7 MB/73.1 MiB) [ 2.881074] vda: detected capacity change from 0 to 76681216 [ 2.896637] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 2.900079] vdb: detected capacity change from 0 to 1073741824 [ 2.907611] libphy: Fixed MDIO Bus: probed [ 2.918078] usbcore: registered new interface driver usbserial_generic [ 2.919901] usbserial: USB Serial support registered for generic [ 2.921703] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 2.925299] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 2.926818] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 2.929041] mousedev: PS/2 mouse device common for all mice [ 2.931922] rtc_cmos 00:05: RTC can wake from S4 [ 2.935820] rtc_cmos 00:05: registered as rtc0 [ 2.937348] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 2.939756] intel_pstate: CPU model not supported [ 2.942046] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 2.946138] hid: raw HID events driver (C) Jiri Kosina [ 2.951835] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 2.952259] usbcore: registered new interface driver usbhid [ 2.957324] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 2.958752] usbhid: USB HID core driver [ 2.958996] drop_monitor: Initializing network drop monitor service [ 2.968502] Initializing XFRM netlink socket [ 2.971381] NET: Registered protocol family 10 [ 2.974841] Segment Routing with IPv6 [ 2.976585] NET: Registered protocol family 17 [ 2.980287] mpls_gso: MPLS GSO support [ 2.986836] RAS: Correctable Errors collector initialized. [ 2.989770] AVX version of gcm_enc/dec engaged. [ 2.991703] AES CTR mode by8 optimization enabled [ 3.067821] sched_clock: Marking stable (3067713494, 0)->(4032056827, -964343333) [ 3.072323] registered taskstats version 1 [ 3.074270] Loading compiled-in X.509 certificates [ 3.077000] zswap: loaded using pool lzo/zbud [ 3.104087] Key type big_key registered [ 3.118249] Key type encrypted registered [ 3.120155] ima: No TPM chip found, activating TPM-bypass! [ 3.122758] ima: Allocated hash algorithm: sha1 [ 3.124686] ima: No architecture policies found [ 3.126663] evm: Initialising EVM extended attributes: [ 3.128675] evm: security.selinux [ 3.130085] evm: security.ima [ 3.131241] evm: security.capability [ 3.132781] evm: HMAC attrs: 0x1 [ 3.135549] rtc_cmos 00:05: setting system clock to 2026-09-05 19:17:57 UTC (1788635877) [ 3.143143] debug: unmapping init [mem 0xffffffffb6403000-0xffffffffb65fffff] [ 3.146722] debug: unmapping init [mem 0xffffffffb5182000-0xffffffffb5458fff] [ 3.155550] Write protecting the kernel read-only data: 28672k [ 3.159487] debug: unmapping init [mem 0xffffffffb3803000-0xffffffffb39fffff] [ 3.162576] debug: unmapping init [mem 0xffffffffb4114000-0xffffffffb41fffff] [ 3.195468] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.205704] systemd[1]: Detected virtualization kvm. [ 3.207556] systemd[1]: Detected architecture x86-64. [ 3.209834] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.235491] systemd[1]: No hostname configured. [ 3.238089] systemd[1]: Set hostname to . [ 3.240980] random: systemd: uninitialized urandom read (16 bytes read) [ 3.244101] systemd[1]: Initializing machine ID from random generator. [ 3.378086] random: systemd: uninitialized urandom read (16 bytes read) [ 3.380751] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 3.385214] random: systemd: uninitialized urandom read (16 bytes read) [ 3.389045] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 3.397587] systemd[1]: Reached target Paths. [ OK ] Reached target Paths. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Slices. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on Journal Socket. [ OK ] Reached target Swap. Starting Journal Service... [ OK ] Started Memstrack Anylazing Service. [ OK ] Listening on udev Control Socket. Starting Apply Kernel Variables... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Sockets. [ OK ] Reached target Local File Systems. Starting Setup Virtual Console... Starting Create Volatile Files and Directories... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 3.984858] device-mapper: uevent: version 1.0.3 [ 3.987128] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] [ 4.709669] random: fast init done Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 4.777580] virtio_net virtio0 ens2: renamed from eth0 [ 4.871860] scsi host0: ata_piix [ 4.948464] scsi host1: ata_piix [ 4.950448] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 4.954792] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 8.664925] dracut-initqueue[579]: RTNETLINK answers: File exists [ 9.641254] random: crng init done [ 9.643726] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.148278] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Timers. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.328558] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.587094] SELinux: Disabled at runtime. [ 11.648609] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.655389] systemd[1]: Detected virtualization kvm. [ 11.656845] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.232297] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.236050] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.242887] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.247571] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.251891] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.258980] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.263682] systemd[1]: Created slice User and Session Slice. [ OK ] Created slice User and Session Slice. Mounting POSIX Message Queue File System... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Starting Remount Root and Kernel File Systems... [ OK ] Reached target rpc_pipefs.target. Starting Apply Kernel Variables... Activating swap /dev/disk/by-label/SWAP... Mounting Huge Pages File System... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on Process Core Dump Socket. [ 12.358407] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Created slice system-getty.slice. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. [ OK ] Reached target Slices. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. Mounting Kernel Debug File System... [ OK ] Started Journal Service. [ OK ] Mounted POSIX Message Queue File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Huge Pages File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Kernel Debug File System. [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt.[ 12.782885] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.170710] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.195708] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.264735] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.281237] EDAC sbridge: Ver: 1.1.2 [ 14.454536] Key type dns_resolver registered [ 14.761515] NFS: Registering the id_resolver key type [ 14.763664] Key type id_resolver registered [ 14.767742] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started irqbalance daemon. Starting Login Service... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... [ OK ] Started GSSAPI Proxy Daemon. Starting Hostname Service... [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. [ OK ] Started Authorization Manager. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg460-client login: [ 52.077574] libcfs: loading out-of-tree module taints kernel. [ 52.194707] Key type ._llcrypt registered [ 52.199814] Key type .llcrypt registered [ 52.964495] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 52.978179] alg: No test for adler32 (adler32-zlib) [ 54.697462] Lustre: Lustre: Build Version: 2.17.58_2_g6837bc1 [ 55.411734] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 57.255421] Key type lgssc registered [ 58.858693] Lustre: Echo OBD driver; http://www.lustre.org/ [ 85.215298] hrtimer: interrupt took 4354128 ns [ 198.102769] Lustre: Mounted lustre-client - version 2.17.58_2_g6837bc1 [ 204.787858] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 220.802113] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing check_logdir /tmp/testlogs/ [ 223.715379] Lustre: lustre-OST0000-osc-ffff88b4989c8000: disconnect after 23s idle [ 226.517748] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing yml_node [ 230.948024] Lustre: DEBUG MARKER: Client: 2.17.58.2 [ 234.049817] Lustre: DEBUG MARKER: MDS: 2.17.58.2 [ 236.755600] Lustre: DEBUG MARKER: OSS: 2.17.58.2 [ 238.471432] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Sat Sep 5 15:21:51 EDT 2026 [ 254.233836] Lustre: Unmounted lustre-client [ 277.808298] Key type lgssc unregistered [ 278.197037] LNet: 8409:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 278.206440] LNetError: 8409:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 278.230219] LNet: Removed LNI 192.168.204.60@tcp [ 279.162165] Key type .llcrypt unregistered [ 279.169214] Key type ._llcrypt unregistered [ 305.777785] Key type ._llcrypt registered [ 305.780493] Key type .llcrypt registered [ 306.015375] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 306.045642] alg: No test for adler32 (adler32-zlib) [ 306.977832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 307.071540] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 307.114751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 312.329257] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 312.456307] LNet: 9281:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 312.471591] LNetError: 9281:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 312.522368] LNet: Removed LNI 192.168.204.60@tcp [ 313.434385] Key type .llcrypt unregistered [ 313.437783] Key type ._llcrypt unregistered [ 319.101970] Key type ._llcrypt registered [ 319.103823] Key type .llcrypt registered [ 320.567543] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 15:23:13 (1788636193) [ 321.082639] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 321.133758] alg: No test for adler32 (adler32-zlib) [ 322.103497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 322.250428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 322.292528] LNet: 9697:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 322.335905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 322.478761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 322.525189] LNet: 9704:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 333.736108] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 15:23:24 (1788636204) [ 333.889645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 333.943467] LNet: 10276:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 334.003297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 334.246984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 344.720779] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 15:23:37 (1788636217) [ 345.119936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 345.168211] LNet: 10870:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 345.219663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 345.351474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 353.838727] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 15:23:46 (1788636226) [ 353.959357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 353.997993] LNet: 11451:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 354.041823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 354.240557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 362.984786] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 15:23:55 (1788636235) [ 363.127137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 363.160156] LNet: 12034:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 363.210439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 363.339788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 363.951430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 373.657693] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 15:24:06 (1788636246) [ 373.914436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 373.982693] LNet: 12629:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 374.051367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 374.139836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 374.838784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 375.599870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 384.477672] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 15:24:17 (1788636257) [ 384.705839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 384.730823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 384.785600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 391.559385] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 15:24:24 (1788636264) [ 391.950164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 392.032490] LNet: 13815:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 392.043848] LNet: 13815:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 392.095384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 392.182606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 392.220317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 392.702940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 392.748960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 393.268237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 393.334670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 393.900499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 393.955491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 394.482702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 394.508502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 394.780527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 394.806698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 395.126165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 395.154921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 395.655878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 395.704260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 396.201506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 396.248829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 396.781125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 396.825528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 397.301914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 397.348195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 397.955754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 398.004991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 398.490729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 398.542892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 406.258365] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 15:24:39 (1788636279) [ 406.488463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 406.522843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 406.620340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 406.657232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 413.774180] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 15:24:46 (1788636286) [ 414.176065] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 414.226378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 414.320756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 414.380235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 421.377798] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 15:24:54 (1788636294) [ 421.557632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 421.588826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 421.684620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 421.737499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 429.568403] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 15:25:02 (1788636302) [ 429.748692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 429.822177] LNet: 16254:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 429.829491] LNet: 16254:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 429.883480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 430.002050] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 430.070404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 437.034954] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 15:25:10 (1788636310) [ 437.240787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 437.291276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 437.338244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 437.368487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 444.219370] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 15:25:17 (1788636317) [ 444.494465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 444.525439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 444.596727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 444.639991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 452.225789] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 15:25:24 (1788636324) [ 452.334970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 452.412940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 452.580533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 453.219330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 460.089196] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 15:25:33 (1788636333) [ 460.321191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 460.376681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 460.708387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 461.041432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 467.469699] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 15:25:40 (1788636340) [ 467.950675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 467.986816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 468.129830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 468.311323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 475.296834] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 15:25:48 (1788636348) [ 475.419118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 475.467737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 475.835797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 476.064548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 483.240348] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 15:25:55 (1788636355) [ 483.356937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 483.417035] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 483.546025] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 483.648789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 491.840249] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 15:26:04 (1788636364) [ 492.290530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 492.339550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 492.488262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 492.557556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 501.595718] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 15:26:14 (1788636374) [ 501.771398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 501.803337] LNet: 21695:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 501.823912] LNet: 21695:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 8 previous similar messages [ 501.914069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 502.072525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 512.183634] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 15:26:24 (1788636384) [ 512.696924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 512.817526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 513.039712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 524.514156] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 15:26:36 (1788636396) [ 525.124910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 525.219905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 525.345646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 525.645903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 533.676695] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 15:26:46 (1788636406) [ 534.026667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 534.070769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 534.187186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 534.238569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 541.902254] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 15:26:54 (1788636414) [ 542.190556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 542.240626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 542.378906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 542.445482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 550.804962] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 15:27:03 (1788636423) [ 551.316056] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 551.425294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 551.500949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 551.553894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 560.046414] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 15:27:12 (1788636432) [ 560.300301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 560.368296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 560.470381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 560.525525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 567.785263] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 15:27:20 (1788636440) [ 568.232612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 568.277439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 568.368333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 568.463281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 568.510217] LNetError: 25821:0:(peer.c:2366:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 568.581269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 568.623831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 574.792171] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 15:27:27 (1788636447) [ 574.989135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 575.061475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 575.186630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.N9WXD9Z3cz [ 575.307475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.N9WXD9Z3cz [ 575.357928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.N9WXD9Z3cz [ 581.193562] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 15:27:34 (1788636454) [ 581.281898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 581.329860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 581.427571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 581.472337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 581.526230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 588.454599] Lustre: DEBUG MARKER: == sanity-lnet test 40: LNetAddPeer merges a pre-existing non-primary peer NID ========================================================== 15:27:41 (1788636461) [ 588.597823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 588.658279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 588.794521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 588.823705] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 590.529726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.60@tcp [ 591.781862] Lustre: Lustre: Build Version: 2.17.58_2_g6837bc1 [ 593.679179] Key type lgssc registered [ 594.762960] Lustre: Echo OBD driver; http://www.lustre.org/ [ 598.074900] LNetError: 27583:0:(lib-move.c:2151:lnet_handle_find_routed_path()) peer 192.168.204.60@tcp99 has no available nets [ 600.416333] Key type lgssc unregistered [ 600.744496] LNetError: 28688:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 600.757197] LNet: Removed LNI 192.168.204.60@tcp [ 601.337127] Key type .llcrypt unregistered [ 601.342945] Key type ._llcrypt unregistered [ 608.997560] Key type ._llcrypt registered [ 609.000951] Key type .llcrypt registered [ 610.178924] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 15:28:03 (1788636483) [ 610.863784] Key type .llcrypt unregistered [ 610.866419] Key type ._llcrypt unregistered [ 617.424609] Key type ._llcrypt registered [ 617.440114] Key type .llcrypt registered [ 617.665906] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 617.678556] alg: No test for adler32 (adler32-zlib) [ 618.617110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 618.694522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 618.725277] LNet: 29598:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=16 less than recommended minimum 512 [ 618.739338] LNet: 29598:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 [ 618.756901] LNet: 29598:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 [ 618.787254] LNet: Message forwarding will be enabled in 2s [ 621.856655] LNet: Message forwarding enabled [ 622.266765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 622.288566] LNet: Message forwarding will stop in 2s to 6s [ 625.969386] LNet: Message forwarding disabled [ 626.831619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 626.855733] LNet: 29696:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=16 less than recommended minimum 512 [ 626.867724] LNet: 29696:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 [ 626.875289] LNet: 29696:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 [ 626.886878] LNet: Message forwarding will be enabled in 2s [ 627.060674] LNet: 29709:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 627.758526] Key type .llcrypt unregistered [ 627.760550] Key type ._llcrypt unregistered [ 640.263580] Key type ._llcrypt registered [ 640.268340] Key type .llcrypt registered [ 641.752306] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 15:28:34 (1788636514) [ 642.102786] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 642.115718] alg: No test for adler32 (adler32-zlib) [ 643.100894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 643.228278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 651.127370] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 15:28:44 (1788636524) [ 651.422852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 651.453471] LNet: 31111:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 651.480715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 651.550616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 651.604811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 651.637372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 651.674417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 651.719326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 651.764799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 651.813730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 651.849336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 651.888184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 651.939985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 651.976570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 652.058629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 652.091988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 652.122712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 652.149694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 652.174541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 652.201493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 652.236113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 652.300536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 652.338909] LNetError: 31154:0:(router.c:625:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 659.935220] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 15:28:52 (1788636532) [ 660.243577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 660.279948] LNet: 31726:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 660.326967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 660.448263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 669.901196] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 15:29:01 (1788636541) [ 672.379055] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg460-server [ 674.490776] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 15:29:07 (1788636547) [ 674.790490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 674.831543] LNet: 32658:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 674.859649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 674.925517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 674.943221] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 676.177803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.204.[61-65]@tcp [ 676.213583] LNet: 32791:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 679.327600] LNetError: 32667:0:(lib-lnet.h:1281:lnet_set_route_aliveness()) route to tcp1 through 192.168.204.62@tcp has gone from up to down [ 682.467243] LNetError: 32666:0:(lib-lnet.h:1281:lnet_set_route_aliveness()) route to tcp1 through 192.168.204.65@tcp has gone from up to down [ 682.490778] LNetError: 32666:0:(lib-lnet.h:1281:lnet_set_route_aliveness()) Skipped 3 previous similar messages [ 683.579654] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 15:29:16 (1788636556) [ 686.039470] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg460-server [ 687.747508] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 15:29:20 (1788636560) [ 687.807466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 687.824636] LNet: 33722:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 687.839698] LNetError: 33722:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 687.867813] LNet: Removed LNI 192.168.204.60@tcp [ 687.896531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 687.961438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 687.976744] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 688.700716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.204.[61-66/2]@tcp [ 688.721781] LNet: 33845:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 688.729956] LNet: 33845:0:(router.c:729:lnet_add_route()) Skipped 4 previous similar messages [ 688.777977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.204.[61-66/2]@tcp [ 695.857536] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 15:29:28 (1788636568) [ 696.194278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 696.209271] LNet: 34438:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 696.219647] LNetError: 34438:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 696.239160] LNet: Removed LNI 192.168.204.60@tcp [ 696.300343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 696.431126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 696.507342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 696.612320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 696.743097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 696.848393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 696.915481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 696.981740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 697.063510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 697.117191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 697.238994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 697.344691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 697.421869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 697.509168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 697.636247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 697.717697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 697.835453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 697.920432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 697.959914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 698.014417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 698.121848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 698.176033] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 698.254863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 698.341947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 698.383557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 698.537496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 698.659264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 698.713151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 698.802354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 698.918755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 698.989883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 699.066245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 699.110770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 699.136670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 699.205851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 707.228735] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 15:29:39 (1788636579) [ 709.598419] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg460-server [ 711.465704] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 15:29:44 (1788636584) [ 713.324642] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg460-server [ 715.159661] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 15:29:48 (1788636588) [ 715.482483] LNet: 35840:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 715.493084] LNet: 35840:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 11 previous similar messages [ 716.114788] Key type .llcrypt unregistered [ 716.116962] Key type ._llcrypt unregistered [ 723.628881] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 723.889317] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 723.895937] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 723.911917] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 726.591102] Key type ._llcrypt registered [ 726.593670] Key type .llcrypt registered [ 726.852255] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 726.884918] alg: No test for adler32 (adler32-zlib) [ 727.907856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 727.970907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 728.004651] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 728.754722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 728.774660] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 730.600341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 730.634780] LNet: Removed LNI 10.1.2.3@tcp [ 730.815441] LNet: 36557:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 730.824489] LNetError: 36557:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 730.839831] LNet: Removed LNI 192.168.204.60@tcp [ 731.497486] Key type .llcrypt unregistered [ 731.499719] Key type ._llcrypt unregistered [ 745.102068] Key type ._llcrypt registered [ 745.107720] Key type .llcrypt registered [ 746.894438] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 15:30:19 (1788636619) [ 747.797463] Key type .llcrypt unregistered [ 747.803050] Key type ._llcrypt unregistered [ 755.677206] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 755.939724] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 755.950301] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 755.958100] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 758.821508] Key type ._llcrypt registered [ 758.825949] Key type .llcrypt registered [ 759.105703] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 759.130369] alg: No test for adler32 (adler32-zlib) [ 760.056182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 760.135178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 760.167405] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 760.870985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 760.891591] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 763.464816] LNet: 38047:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 763.469947] LNetError: 38047:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 763.477408] LNet: Removed LNI 10.1.2.3@tcp [ 763.486759] LNet: Removed LNI 192.168.204.60@tcp [ 764.124895] Key type .llcrypt unregistered [ 764.128657] Key type ._llcrypt unregistered [ 779.294991] Key type ._llcrypt registered [ 779.297843] Key type .llcrypt registered [ 780.825841] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 15:30:53 (1788636653) [ 782.090527] Key type .llcrypt unregistered [ 782.092660] Key type ._llcrypt unregistered [ 790.661936] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 790.952450] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 790.956120] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 790.964095] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 793.545471] Key type ._llcrypt registered [ 793.548928] Key type .llcrypt registered [ 793.768969] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 793.781713] alg: No test for adler32 (adler32-zlib) [ 794.698211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 794.783228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 794.820491] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 797.259661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 797.284949] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 800.120695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 800.137291] LNet: Removed LNI 10.1.2.3@tcp [ 800.150946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 800.187352] LNet: Removed LNI 10.1.2.31@tcp [ 800.202719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 800.231163] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 801.073893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 801.105760] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 802.534298] LNet: 39959:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 802.543657] LNetError: 39959:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg:0 not found [ 802.553748] LNet: Removed LNI 10.1.2.31@tcp [ 802.574432] LNet: Removed LNI 10.1.2.3@tcp [ 803.410560] Key type .llcrypt unregistered [ 803.415235] Key type ._llcrypt unregistered [ 817.849990] Key type ._llcrypt registered [ 817.860583] Key type .llcrypt registered [ 819.885940] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 15:31:31 (1788636691) [ 820.374584] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 820.393923] alg: No test for adler32 (adler32-zlib) [ 821.334096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 821.383216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 821.399230] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 821.972493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 821.987861] LNet: 40910:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 821.993720] LNetError: 40910:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 822.013269] LNet: Removed LNI 192.168.204.60@tcp [ 822.067060] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 822.134942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.204.60@tcp [ 822.171390] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 822.846988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.204.60@tcp [ 822.903691] LNet: Removed LNI 192.168.204.60@tcp [ 822.998540] LNet: 41045:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 823.681479] Key type .llcrypt unregistered [ 823.684657] Key type ._llcrypt unregistered [ 836.643567] Key type ._llcrypt registered [ 836.647261] Key type .llcrypt registered [ 838.207698] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 15:31:50 (1788636710) [ 840.178471] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg460-server [ 842.060565] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 15:31:54 (1788636714) [ 842.896498] Key type .llcrypt unregistered [ 842.901264] Key type ._llcrypt unregistered [ 850.256403] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 850.447738] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 850.455819] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 850.472708] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 853.298148] Key type ._llcrypt registered [ 853.300178] Key type .llcrypt registered [ 853.557517] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 853.588870] alg: No test for adler32 (adler32-zlib) [ 854.580434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 854.674198] LNet: Added LNI 192.168.204.60@tcp1 [8/256/0/180] [ 855.761983] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 858.244419] LNet: 42875:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 858.249888] LNetError: 42875:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 858.259528] LNet: Removed LNI 192.168.204.60@tcp1 [ 858.267268] LNet: Removed LNI 10.1.2.3@tcp2 [ 858.944393] Key type .llcrypt unregistered [ 858.945567] Key type ._llcrypt unregistered [ 873.683075] Key type ._llcrypt registered [ 873.687039] Key type .llcrypt registered [ 875.234450] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 15:32:27 (1788636747) [ 875.579663] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 875.590867] alg: No test for adler32 (adler32-zlib) [ 876.532996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 876.591957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 876.612846] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 877.511315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 877.536681] LNet: 43842:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 877.543813] LNetError: 43842:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 877.562383] LNet: Removed LNI 192.168.204.60@tcp [ 877.594129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 877.691609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 877.743963] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 885.220738] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 15:32:38 (1788636758) [ 885.619489] LNet: 44433:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 885.634988] LNetError: 44433:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 885.672385] LNet: Removed LNI 192.168.204.60@tcp [ 886.428733] Key type .llcrypt unregistered [ 886.430230] Key type ._llcrypt unregistered [ 893.679431] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 893.876581] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 893.885928] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 893.891298] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 896.549306] Key type ._llcrypt registered [ 896.552082] Key type .llcrypt registered [ 896.782400] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 896.817654] alg: No test for adler32 (adler32-zlib) [ 897.788876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 897.894145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 897.922799] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 898.767356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 898.788842] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 901.055372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 901.076410] LNet: 45172:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 901.083993] LNetError: 45172:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 901.091284] LNet: Removed LNI 10.1.2.3@tcp [ 901.105486] LNet: Removed LNI 192.168.204.60@tcp [ 901.130687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 901.193689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 901.237299] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 901.243940] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 901.930052] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 901.968864] LNet: 45199:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 901.978969] LNetError: 45199:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 901.986237] LNetError: 45199:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 901.999760] LNet: Removed LNI 10.1.2.3@tcp [ 902.012913] LNet: Removed LNI 192.168.204.60@tcp [ 902.041710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 902.097727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 902.133970] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 902.139818] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 902.815759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 902.848875] LNet: Added LNI 192.168.204.60@tcp2 [8/256/0/180] [ 904.042803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 904.064811] LNet: 45362:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 904.069831] LNetError: 45362:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 904.077774] LNetError: 45362:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 904.084645] LNet: Removed LNI 10.1.2.3@tcp [ 904.087906] LNet: Removed LNI 192.168.204.60@tcp [ 904.100635] LNet: Removed LNI 192.168.204.60@tcp2 [ 904.114965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 904.172868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 904.189767] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 904.195864] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 904.203161] LNet: Added LNI 192.168.204.60@tcp2 [8/256/0/180] [ 912.681108] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 15:33:05 (1788636785) [ 912.922431] LNet: 45960:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 912.948191] LNetError: 45960:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 912.964580] LNetError: 45960:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 912.979852] LNet: Removed LNI 10.1.2.3@tcp [ 912.982654] LNet: Removed LNI 192.168.204.60@tcp [ 913.008275] LNet: Removed LNI 192.168.204.60@tcp2 [ 913.805720] Key type .llcrypt unregistered [ 913.810417] Key type ._llcrypt unregistered [ 922.889748] Key type ._llcrypt registered [ 922.891656] Key type .llcrypt registered [ 923.035074] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 923.051147] alg: No test for adler32 (adler32-zlib) [ 924.040620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 924.182368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 924.244125] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 925.487118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 925.514205] LNet: 46367:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 925.521495] LNetError: 46367:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 925.541714] LNet: Removed LNI 192.168.204.60@tcp [ 925.567280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 925.637355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 925.665255] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 926.549475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 926.570399] LNet: 46397:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 926.576507] LNetError: 46397:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 926.598749] LNet: Removed LNI 192.168.204.60@tcp [ 926.622227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 926.710365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 926.732466] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 927.284457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 927.316918] LNet: Removed LNI 192.168.204.60@tcp [ 927.371403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 927.463273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 927.491741] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 927.906932] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 928.113119] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 928.129759] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 928.150667] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 930.605800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 930.625834] LNet: 46501:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 930.630764] LNet: 46501:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 930.636666] LNetError: 46501:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 930.641734] LNetError: 46501:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 930.665789] LNet: Removed LNI 192.168.204.60@tcp [ 930.684316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 930.815427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 930.860427] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 939.443605] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 15:33:32 (1788636812) [ 939.856701] LNet: 47099:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 939.865687] LNetError: 47099:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 939.881499] LNet: Removed LNI 192.168.204.60@tcp [ 940.698049] Key type .llcrypt unregistered [ 940.708541] Key type ._llcrypt unregistered [ 950.098805] Key type ._llcrypt registered [ 950.108189] Key type .llcrypt registered [ 950.384057] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 950.398695] alg: No test for adler32 (adler32-zlib) [ 951.404196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 951.531781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 951.596472] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 952.533780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 952.578706] LNet: 47495:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 952.594073] LNetError: 47495:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 952.626685] LNet: Removed LNI 192.168.204.60@tcp [ 952.682852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 952.775952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 952.827784] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 960.143253] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 15:33:53 (1788636833) [ 960.273687] LNet: 48079:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 960.289417] LNetError: 48079:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 960.311376] LNet: Removed LNI 192.168.204.60@tcp [ 961.351755] Key type .llcrypt unregistered [ 961.357113] Key type ._llcrypt unregistered [ 968.102742] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 968.321540] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 968.336554] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 968.339983] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 969.874618] Key type ._llcrypt registered [ 969.877576] Key type .llcrypt registered [ 970.023725] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 970.038980] alg: No test for adler32 (adler32-zlib) [ 970.919581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 970.964936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 970.982231] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 971.856660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 971.882480] LNet: 48525:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 971.899959] LNetError: 48525:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 971.925381] LNet: Removed LNI 192.168.204.60@tcp [ 971.968969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 972.049128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 972.091805] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 980.878674] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 15:34:13 (1788636853) [ 981.011683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 981.067727] LNet: 49120:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 981.084897] LNetError: 49120:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 981.116705] LNet: Removed LNI 192.168.204.60@tcp [ 981.205683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 981.402892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 988.296076] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 15:34:21 (1788636861) [ 988.887877] LNet: 49706:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 989.619587] Key type .llcrypt unregistered [ 989.622754] Key type ._llcrypt unregistered [ 997.246730] Key type ._llcrypt registered [ 997.253684] Key type .llcrypt registered [ 997.430728] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 997.445360] alg: No test for adler32 (adler32-zlib) [ 998.375308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 998.411375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 998.432758] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 999.386543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 999.427866] LNet: 50100:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 999.440358] LNetError: 50100:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 999.454942] LNet: Removed LNI 192.168.204.60@tcp [ 999.480479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 999.551655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 999.585752] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 1007.290035] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 15:34:40 (1788636880) [ 1007.541638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1007.578203] LNet: 50694:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1007.588065] LNetError: 50694:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1007.614639] LNet: Removed LNI 192.168.204.60@tcp [ 1007.640164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1007.714946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 1013.652963] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 15:34:46 (1788636886) [ 1015.284293] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 1017.317735] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 15:34:50 (1788636890) [ 1019.026707] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 1021.550970] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 15:34:54 (1788636894) [ 1023.116542] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 1025.553124] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 15:34:58 (1788636898) [ 1027.320510] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 1029.468745] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 15:35:02 (1788636902) [ 1031.113446] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 1033.434605] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing ========================================================== 15:35:06 (1788636906) [ 1033.948263] LNet: 52748:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1034.754805] Key type .llcrypt unregistered [ 1034.762029] Key type ._llcrypt unregistered [ 1041.751125] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1041.920806] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1041.933801] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1041.940559] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1044.605144] Key type ._llcrypt registered [ 1044.616447] Key type .llcrypt registered [ 1044.890666] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1044.910471] alg: No test for adler32 (adler32-zlib) [ 1045.836649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1045.910650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1045.927948] LNet: Added LNI 192.168.204.60@tcp [8/512/0/180] [ 1045.933781] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 1045.937318] LNet: 53066:0:(api-ni.c:2991:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1046.072672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1046.098702] LNet: 53085:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1046.114057] LNetError: 53085:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1046.128132] LNet: Removed LNI 10.1.2.3@tcp [ 1046.129827] LNetError: 53085:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1046.144481] LNet: Removed LNI 192.168.204.60@tcp [ 1046.171630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1046.244233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1046.263815] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 1046.334175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1046.358693] LNet: 53109:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1046.365826] LNetError: 53109:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1046.384864] LNet: Removed LNI 192.168.204.60@tcp [ 1046.443406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1046.545343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1046.577805] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 1046.587283] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1046.589285] LNet: 53117:0:(api-ni.c:2991:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1046.687965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1046.708817] LNet: 53135:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1046.716375] LNetError: 53135:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1046.722351] LNet: Removed LNI 10.1.2.3@tcp [ 1046.724386] LNetError: 53135:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1046.739542] LNet: Removed LNI 192.168.204.60@tcp [ 1046.773706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1046.872592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1046.915040] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 1046.923118] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1046.927916] LNet: 53143:0:(api-ni.c:2991:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1047.262821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1047.302880] LNet: 53169:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1047.309359] LNetError: 53169:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1047.317234] LNet: Removed LNI 10.1.2.3@tcp [ 1047.331615] LNetError: 53169:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1047.351170] LNet: Removed LNI 192.168.204.60@tcp [ 1047.388609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1047.501328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1047.528519] LNet: Added LNI 192.168.204.60@tcp [8/256/0/182] [ 1047.546467] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 1047.709272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1047.734430] LNet: 53197:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1047.745896] LNetError: 53197:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1047.765602] LNet: Removed LNI 10.1.2.3@tcp [ 1047.767214] LNetError: 53197:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1047.797662] LNet: Removed LNI 192.168.204.60@tcp [ 1047.821687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1047.914952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1047.950856] LNet: Added LNI 192.168.204.60@tcp [8/256/0/182] [ 1047.956874] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 1047.960310] LNet: 53205:0:(api-ni.c:2991:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1048.047583] LNet: Removed LNI 10.1.2.3@tcp [ 1048.058297] LNet: Removed LNI 192.168.204.60@tcp [ 1048.778705] Key type .llcrypt unregistered [ 1048.780425] Key type ._llcrypt unregistered [ 1061.401030] Key type ._llcrypt registered [ 1061.403554] Key type .llcrypt registered [ 1063.238279] Lustre: DEBUG MARKER: == sanity-lnet test 180: FastReg map error should not leak descriptors ========================================================== 15:35:35 (1788636935) [ 1065.371798] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_180 Need o2ib NETTYPE [ 1067.529535] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 15:35:40 (1788636940) [ 1068.438744] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1068.480456] alg: No test for adler32 (adler32-zlib) [ 1069.399814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1069.481511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1069.512384] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 1070.380994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.204.60 [ 1070.404353] LNet: 54467:0:(config.c:1740:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1070.420853] LNet: Removed LNI 192.168.204.60@tcp [ 1078.193440] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 15:35:50 (1788636950) [ 1078.333329] LNet: 55035:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1078.339867] LNetError: 55035:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1078.374500] LNet: Removed LNI 192.168.204.60@tcp [ 1079.186485] Key type .llcrypt unregistered [ 1079.189850] Key type ._llcrypt unregistered [ 1087.464176] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1087.780808] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1090.753781] Key type ._llcrypt registered [ 1090.760728] Key type .llcrypt registered [ 1091.063680] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1091.094072] alg: No test for adler32 (adler32-zlib) [ 1092.657602] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1100.174915] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 15:36:13 (1788636973) [ 1100.595492] LNet: 55950:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1100.604241] LNetError: 55950:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1100.626289] LNet: Removed LNI 10.1.2.3@tcp [ 1101.294747] Key type .llcrypt unregistered [ 1101.298091] Key type ._llcrypt unregistered [ 1108.567123] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1108.863280] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1108.868353] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1108.871652] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1111.840435] Key type ._llcrypt registered [ 1111.842605] Key type .llcrypt registered [ 1112.078878] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1112.110881] alg: No test for adler32 (adler32-zlib) [ 1113.139657] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1121.450747] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 15:36:34 (1788636994) [ 1122.924655] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 1125.432599] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 15:36:38 (1788636998) [ 1125.792264] LNet: 57132:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1125.804950] LNetError: 57132:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1125.825062] LNet: Removed LNI 10.1.2.3@tcp [ 1126.660394] Key type .llcrypt unregistered [ 1126.665416] Key type ._llcrypt unregistered [ 1134.352289] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1134.648711] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1134.659791] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1134.671636] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1137.455401] Key type ._llcrypt registered [ 1137.458280] Key type .llcrypt registered [ 1137.705168] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1137.729668] alg: No test for adler32 (adler32-zlib) [ 1138.766108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1138.848186] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1148.903665] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 15:37:01 (1788637021) [ 1149.492637] LNet: 58358:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1149.500239] LNetError: 58358:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1149.532213] LNet: Removed LNI 10.1.2.3@tcp [ 1150.304588] Key type .llcrypt unregistered [ 1150.306468] Key type ._llcrypt unregistered [ 1158.831770] Key type ._llcrypt registered [ 1158.834698] Key type .llcrypt registered [ 1159.114213] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1159.133347] alg: No test for adler32 (adler32-zlib) [ 1160.125139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1160.163923] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 1170.647933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.160@tcp [ 1171.529536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.60@tcp -d 192.168.204.160@tcp -m GET -e local_interrupt -r 1 [ 1171.580935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.60@tcp -d 192.168.204.60@tcp -m GET -e local_interrupt -r 1 [ 1171.645298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.160@tcp [ 1171.687419] LNet: There was an unexpected network error while writing to 192.168.204.160: rc = -22 [ 1171.987643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1172.027458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1172.055305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1172.087040] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1172.144308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1172.188700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1172.218138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1172.253915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1172.296212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1172.319333] LNet: 58795:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1172.349511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1172.403791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1172.436306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1172.461867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1172.494793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1172.516331] LNet: 1 local NIs in recovery (showing 1): 192.168.204.60@tcp [ 1172.524402] LNet: There was an unexpected network error while writing to 192.168.204.60: rc = -22 [ 1172.526426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1172.553758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1172.984623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.60@tcp -d 192.168.204.160@tcp -m GET -e local_dropped -r 1 [ 1173.012248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.60@tcp -d 192.168.204.60@tcp -m GET -e local_dropped -r 1 [ 1173.067856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.160@tcp [ 1173.339027] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1173.364732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1173.391586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1173.439571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1173.479386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1173.536745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1173.567541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1173.616957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1173.654676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1173.671766] LNet: 58907:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1173.688555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1173.726499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1173.779658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1173.807994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1173.874482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1173.910376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1173.952000] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1174.800445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.60@tcp -d 192.168.204.160@tcp -m GET -e local_aborted -r 1 [ 1174.829960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.60@tcp -d 192.168.204.60@tcp -m GET -e local_aborted -r 1 [ 1174.881920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.160@tcp [ 1174.900542] LNet: There was an unexpected network error while writing to 192.168.204.160: rc = -22 [ 1174.907355] LNet: Skipped 1 previous similar message [ 1175.185732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1175.231701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1175.281793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1175.349148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1175.404429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1175.472897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1175.528289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1175.592766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1175.645661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1175.659611] LNet: 59018:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1175.696305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1175.759036] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1175.794518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1175.848499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1175.913227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1175.979453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1176.063662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1176.903563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.60@tcp -d 192.168.204.160@tcp -m GET -e local_no_route -r 1 [ 1176.961549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.60@tcp -d 192.168.204.60@tcp -m GET -e local_no_route -r 1 [ 1177.023450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.160@tcp [ 1177.055282] LNet: There was an unexpected network error while writing to 192.168.204.160: rc = -22 [ 1177.353878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1177.382703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1177.414461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1177.444868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1177.486624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1177.522534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1177.552614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1177.612812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1177.637537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1177.665059] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1177.710389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1177.748972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1177.813891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1177.882513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1177.962717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1178.023531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1178.700262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.60@tcp -d 192.168.204.160@tcp -m GET -e local_timeout -r 1 [ 1178.769425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.60@tcp -d 192.168.204.60@tcp -m GET -e local_timeout -r 1 [ 1178.845731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.160@tcp [ 1179.175281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1179.205561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1179.228323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1179.255221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1179.288520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1179.334039] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1179.394983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1179.459407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1179.505984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1179.525742] LNet: 59240:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1179.534848] LNet: 59240:0:(api-ni.c:301:recovery_interval_set()) Skipped 1 previous similar message [ 1179.550328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1179.577495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1179.619454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1179.666200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1179.731189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1179.772852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1179.832650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1180.422127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.60@tcp -d 192.168.204.160@tcp -m GET -e local_error -r 1 [ 1180.454180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.60@tcp -d 192.168.204.60@tcp -m GET -e local_error -r 1 [ 1180.484209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.160@tcp [ 1180.742503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1180.773774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1180.837281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1180.886938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1180.937969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1180.994324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1181.053823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1181.130026] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1181.179619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1181.268479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1181.313686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1181.367873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1181.416235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1181.466270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1181.520516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1181.592908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1182.008053] LNet: 59393:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1182.020302] LNetError: 59393:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1182.059098] LNet: Removed LNI 192.168.204.60@tcp [ 1182.548932] Key type .llcrypt unregistered [ 1182.550994] Key type ._llcrypt unregistered [ 1203.051152] Key type ._llcrypt registered [ 1203.053064] Key type .llcrypt registered [ 1204.459958] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 15:37:57 (1788637077) [ 1205.577348] Key type .llcrypt unregistered [ 1205.580123] Key type ._llcrypt unregistered [ 1212.881706] Key type ._llcrypt registered [ 1212.887090] Key type .llcrypt registered [ 1213.146473] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1213.169815] alg: No test for adler32 (adler32-zlib) [ 1214.137268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1214.192528] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 1224.003388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.160@tcp [ 1238.468236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1238.565872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1238.603522] LNet: Added LNI 192.168.204.60@tcp1 [8/256/0/180] [ 1239.957634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.60@tcp -d 192.168.204.160@tcp -m GET -e local_interrupt -r 1 [ 1240.044579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.60@tcp -d 192.168.204.160@tcp1 -m GET -e local_interrupt -r 1 [ 1240.145303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.60@tcp -d 192.168.204.60@tcp -m GET -e local_interrupt -r 1 [ 1240.183785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.60@tcp -d 192.168.204.60@tcp1 -m GET -e local_interrupt -r 1 [ 1240.254407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.60@tcp1 -d 192.168.204.160@tcp -m GET -e local_interrupt -r 1 [ 1240.301855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.60@tcp1 -d 192.168.204.160@tcp1 -m GET -e local_interrupt -r 1 [ 1240.338634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.60@tcp1 -d 192.168.204.60@tcp -m GET -e local_interrupt -r 1 [ 1240.398278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.60@tcp1 -d 192.168.204.60@tcp1 -m GET -e local_interrupt -r 1 [ 1240.501537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.160@tcp [ 1240.541781] LNet: There was an unexpected network error while writing to 192.168.204.160: rc = -22 [ 1240.554028] LNet: 1 local NIs in recovery (showing 1): 192.168.204.60@tcp [ 1240.848948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1240.881315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1240.916218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1240.968555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1241.027745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1241.099595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1241.150609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1241.213747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1241.279135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1241.316848] LNet: 60859:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1241.350221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1241.404315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1241.443072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1241.508023] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1241.558897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1241.568746] LNet: There was an unexpected network error while writing to 192.168.204.60: rc = -22 [ 1241.602441] LNet: Skipped 3 previous similar messages [ 1241.644940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1241.697163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1242.321422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.60@tcp -d 192.168.204.160@tcp -m GET -e local_dropped -r 1 [ 1242.362585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.60@tcp -d 192.168.204.160@tcp1 -m GET -e local_dropped -r 1 [ 1242.448558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.60@tcp -d 192.168.204.60@tcp -m GET -e local_dropped -r 1 [ 1242.487510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.60@tcp -d 192.168.204.60@tcp1 -m GET -e local_dropped -r 1 [ 1242.527381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.60@tcp1 -d 192.168.204.160@tcp -m GET -e local_dropped -r 1 [ 1242.570297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.60@tcp1 -d 192.168.204.160@tcp1 -m GET -e local_dropped -r 1 [ 1242.616964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.60@tcp1 -d 192.168.204.60@tcp -m GET -e local_dropped -r 1 [ 1242.665088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.60@tcp1 -d 192.168.204.60@tcp1 -m GET -e local_dropped -r 1 [ 1242.723388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.160@tcp [ 1242.752988] LNet: There was an unexpected network error while writing to 192.168.204.160: rc = -22 [ 1243.158378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1243.215305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1243.268386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1243.357426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1243.403110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1243.431779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1243.458846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1243.504537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1243.551404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1243.567360] LNet: 60983:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1243.586324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1243.626542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1243.691718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1243.728625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1243.803240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1243.860176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1243.928708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1244.659319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.60@tcp -d 192.168.204.160@tcp -m GET -e local_aborted -r 1 [ 1244.737133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.60@tcp -d 192.168.204.160@tcp1 -m GET -e local_aborted -r 1 [ 1244.780803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.60@tcp -d 192.168.204.60@tcp -m GET -e local_aborted -r 1 [ 1244.812399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.60@tcp -d 192.168.204.60@tcp1 -m GET -e local_aborted -r 1 [ 1244.846497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.60@tcp1 -d 192.168.204.160@tcp -m GET -e local_aborted -r 1 [ 1244.883710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.60@tcp1 -d 192.168.204.160@tcp1 -m GET -e local_aborted -r 1 [ 1244.916299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.60@tcp1 -d 192.168.204.60@tcp -m GET -e local_aborted -r 1 [ 1244.959829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.60@tcp1 -d 192.168.204.60@tcp1 -m GET -e local_aborted -r 1 [ 1245.012828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.160@tcp [ 1245.027812] LNet: There was an unexpected network error while writing to 192.168.204.160: rc = -22 [ 1245.032310] LNet: Skipped 2 previous similar messages [ 1245.236117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1245.270557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1245.303487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1245.353945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1245.380267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1245.398856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1245.420600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1245.462717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1245.499349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1245.515307] LNet: 61106:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1245.529362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1245.574149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1245.626587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1245.653267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1245.687244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1245.714903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1245.742776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1246.451883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.60@tcp -d 192.168.204.160@tcp -m GET -e local_no_route -r 1 [ 1246.509982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.60@tcp -d 192.168.204.160@tcp1 -m GET -e local_no_route -r 1 [ 1246.545841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.60@tcp -d 192.168.204.60@tcp -m GET -e local_no_route -r 1 [ 1246.586773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.60@tcp -d 192.168.204.60@tcp1 -m GET -e local_no_route -r 1 [ 1246.637690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.60@tcp1 -d 192.168.204.160@tcp -m GET -e local_no_route -r 1 [ 1246.688759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.60@tcp1 -d 192.168.204.160@tcp1 -m GET -e local_no_route -r 1 [ 1246.761446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.60@tcp1 -d 192.168.204.60@tcp -m GET -e local_no_route -r 1 [ 1246.807242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.60@tcp1 -d 192.168.204.60@tcp1 -m GET -e local_no_route -r 1 [ 1246.842874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.160@tcp [ 1247.172970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1247.237962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1247.310393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1247.401700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1247.458357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1247.506683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1247.548208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1247.596177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1247.652895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1247.690448] LNet: 61229:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1247.745236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1247.822291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1247.883237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1247.922061] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1247.960708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1248.015466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1248.039605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1248.490300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.60@tcp -d 192.168.204.160@tcp -m GET -e local_timeout -r 1 [ 1248.533462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.60@tcp -d 192.168.204.160@tcp1 -m GET -e local_timeout -r 1 [ 1248.597888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.60@tcp -d 192.168.204.60@tcp -m GET -e local_timeout -r 1 [ 1248.652450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.60@tcp -d 192.168.204.60@tcp1 -m GET -e local_timeout -r 1 [ 1248.721850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.60@tcp1 -d 192.168.204.160@tcp -m GET -e local_timeout -r 1 [ 1248.776109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.60@tcp1 -d 192.168.204.160@tcp1 -m GET -e local_timeout -r 1 [ 1248.846547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.60@tcp1 -d 192.168.204.60@tcp -m GET -e local_timeout -r 1 [ 1248.889430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.60@tcp1 -d 192.168.204.60@tcp1 -m GET -e local_timeout -r 1 [ 1248.953476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.160@tcp [ 1249.264266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1249.296836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1249.325839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1249.367531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1249.428922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1249.481796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1249.559625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1249.609119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1249.655375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1249.716693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1249.774691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1249.843043] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1249.898585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1249.962438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1249.999161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1250.024918] LNet: There was an unexpected network error while writing to 192.168.204.60: rc = -22 [ 1250.029763] LNet: Skipped 9 previous similar messages [ 1250.047458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1250.736343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.60@tcp -d 192.168.204.160@tcp -m GET -e local_error -r 1 [ 1250.828726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.60@tcp -d 192.168.204.160@tcp1 -m GET -e local_error -r 1 [ 1250.943789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.60@tcp -d 192.168.204.60@tcp -m GET -e local_error -r 1 [ 1250.993543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.60@tcp -d 192.168.204.60@tcp1 -m GET -e local_error -r 1 [ 1251.054439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.60@tcp1 -d 192.168.204.160@tcp -m GET -e local_error -r 1 [ 1251.112505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.60@tcp1 -d 192.168.204.160@tcp1 -m GET -e local_error -r 1 [ 1251.174411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.60@tcp1 -d 192.168.204.60@tcp -m GET -e local_error -r 1 [ 1251.237445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.60@tcp1 -d 192.168.204.60@tcp1 -m GET -e local_error -r 1 [ 1251.329755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.160@tcp [ 1251.650786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1251.694350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1251.765075] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1251.816080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1251.839349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1251.875355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1251.928102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1251.969681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1252.011900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1252.031324] LNet: 61476:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1252.040144] LNet: 61476:0:(api-ni.c:301:recovery_interval_set()) Skipped 1 previous similar message [ 1252.060971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1252.099897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1252.154253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1252.192235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1252.283961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1252.332501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1252.369263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1259.009230] LNet: 61552:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1259.029049] LNetError: 61552:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1259.038733] LNet: Removed LNI 192.168.204.60@tcp [ 1260.070495] LNet: Removed LNI 192.168.204.60@tcp1 [ 1260.712653] Key type .llcrypt unregistered [ 1260.717069] Key type ._llcrypt unregistered [ 1279.816509] Key type ._llcrypt registered [ 1279.818626] Key type .llcrypt registered [ 1281.551743] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 15:39:13 (1788637153) [ 1282.567402] Key type .llcrypt unregistered [ 1282.570814] Key type ._llcrypt unregistered [ 1290.839763] Key type ._llcrypt registered [ 1290.841914] Key type .llcrypt registered [ 1291.109784] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1291.146927] alg: No test for adler32 (adler32-zlib) [ 1292.108907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1292.143233] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 1300.675590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.160@tcp [ 1301.662879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.60@tcp -d 192.168.204.160@tcp -m GET -e remote_dropped -r 1 [ 1301.743891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.60@tcp -d 192.168.204.60@tcp -m GET -e remote_dropped -r 1 [ 1301.793326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.160@tcp [ 1301.832446] LNet: There was an unexpected network error while writing to 192.168.204.160: rc = -22 [ 1302.088173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1302.119809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1302.145236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1302.170294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1302.197995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1302.223151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1302.247420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1302.271915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1302.296771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1302.310770] LNet: 62798:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1302.341090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1302.367896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1302.391636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1302.425786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1302.469405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1302.519168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1302.551574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1303.128090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.60@tcp -d 192.168.204.160@tcp -m GET -e remote_error -r 1 [ 1303.180521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.60@tcp -d 192.168.204.60@tcp -m GET -e remote_error -r 1 [ 1303.247512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.160@tcp [ 1303.302981] LNet: There was an unexpected network error while writing to 192.168.204.160: rc = -22 [ 1303.575975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1303.627828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1303.681613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1303.775785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1303.833919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1303.886581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1303.959187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1304.008634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1304.083270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1304.099646] LNet: 62909:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1304.125909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1304.167880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1304.235686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1304.289362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1304.333536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1304.371242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1304.429506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1305.068695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.60@tcp -d 192.168.204.160@tcp -m GET -e remote_timeout -r 1 [ 1305.127658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.60@tcp -d 192.168.204.60@tcp -m GET -e remote_timeout -r 1 [ 1305.205950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.160@tcp [ 1305.253739] LNet: There was an unexpected network error while writing to 192.168.204.160: rc = -22 [ 1305.555653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1305.603044] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1305.671594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1305.736423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1305.792653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1305.839362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1305.883372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1305.937545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1306.004461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1306.046374] LNet: 63021:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1306.068528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1306.136299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1306.200495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1306.243850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1306.287177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1306.379116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1306.431453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1306.811723] LNet: 63062:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1306.826769] LNetError: 63062:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1306.870416] LNet: Removed LNI 192.168.204.60@tcp [ 1307.385511] Key type .llcrypt unregistered [ 1307.388763] Key type ._llcrypt unregistered [ 1324.591635] Key type ._llcrypt registered [ 1324.593622] Key type .llcrypt registered [ 1326.091521] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 15:39:58 (1788637198) [ 1327.590921] Key type .llcrypt unregistered [ 1327.596204] Key type ._llcrypt unregistered [ 1335.658492] Key type ._llcrypt registered [ 1335.666865] Key type .llcrypt registered [ 1335.911866] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1335.927440] alg: No test for adler32 (adler32-zlib) [ 1336.796388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1336.827813] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 1346.578175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.160@tcp [ 1355.612886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1355.685111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1355.737113] LNet: Added LNI 192.168.204.60@tcp1 [8/256/0/180] [ 1357.136746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.60@tcp -d 192.168.204.160@tcp -m GET -e remote_dropped -r 1 [ 1357.171251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.60@tcp -d 192.168.204.160@tcp1 -m GET -e remote_dropped -r 1 [ 1357.205877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.60@tcp -d 192.168.204.60@tcp -m GET -e remote_dropped -r 1 [ 1357.238313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.60@tcp -d 192.168.204.60@tcp1 -m GET -e remote_dropped -r 1 [ 1357.282985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.60@tcp1 -d 192.168.204.160@tcp -m GET -e remote_dropped -r 1 [ 1357.334622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.60@tcp1 -d 192.168.204.160@tcp1 -m GET -e remote_dropped -r 1 [ 1357.370887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.60@tcp1 -d 192.168.204.60@tcp -m GET -e remote_dropped -r 1 [ 1357.426459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.60@tcp1 -d 192.168.204.60@tcp1 -m GET -e remote_dropped -r 1 [ 1357.473824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.160@tcp [ 1357.525475] LNet: There was an unexpected network error while writing to 192.168.204.160: rc = -22 [ 1357.532616] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.160@tcp [ 1357.830263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1357.859497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1357.909825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1357.962643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1358.001309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1358.029050] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1358.057721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1358.090784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1358.128173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1358.173250] LNet: 64525:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1358.216472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1358.265751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1358.307410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1358.340334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1358.363734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1358.392622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1358.431460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1358.454035] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1358.844589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.60@tcp -d 192.168.204.160@tcp -m GET -e remote_error -r 1 [ 1358.880991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.60@tcp -d 192.168.204.160@tcp1 -m GET -e remote_error -r 1 [ 1358.937760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.60@tcp -d 192.168.204.60@tcp -m GET -e remote_error -r 1 [ 1359.006616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.60@tcp -d 192.168.204.60@tcp1 -m GET -e remote_error -r 1 [ 1359.054719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.60@tcp1 -d 192.168.204.160@tcp -m GET -e remote_error -r 1 [ 1359.104321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.60@tcp1 -d 192.168.204.160@tcp1 -m GET -e remote_error -r 1 [ 1359.160909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.60@tcp1 -d 192.168.204.60@tcp -m GET -e remote_error -r 1 [ 1359.214937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.60@tcp1 -d 192.168.204.60@tcp1 -m GET -e remote_error -r 1 [ 1359.275858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.160@tcp [ 1359.298795] LNet: There was an unexpected network error while writing to 192.168.204.160: rc = -22 [ 1359.303112] LNet: Skipped 2 previous similar messages [ 1359.510231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1359.548822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1359.596551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1359.643409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1359.689619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1359.734371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1359.773638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1359.801837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1359.843596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1359.861509] LNet: 64651:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1359.887365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1359.936253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1359.968305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1360.004387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1360.040760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1360.061524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1360.083693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1360.109410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1360.719436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.60@tcp -d 192.168.204.160@tcp -m GET -e remote_timeout -r 1 [ 1360.766431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.60@tcp -d 192.168.204.160@tcp1 -m GET -e remote_timeout -r 1 [ 1360.796324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.60@tcp -d 192.168.204.60@tcp -m GET -e remote_timeout -r 1 [ 1360.841309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.60@tcp -d 192.168.204.60@tcp1 -m GET -e remote_timeout -r 1 [ 1360.886209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.60@tcp1 -d 192.168.204.160@tcp -m GET -e remote_timeout -r 1 [ 1360.935738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.60@tcp1 -d 192.168.204.160@tcp1 -m GET -e remote_timeout -r 1 [ 1360.996888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.60@tcp1 -d 192.168.204.60@tcp -m GET -e remote_timeout -r 1 [ 1361.066132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.60@tcp1 -d 192.168.204.60@tcp1 -m GET -e remote_timeout -r 1 [ 1361.131348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.160@tcp [ 1361.166865] LNet: There was an unexpected network error while writing to 192.168.204.160: rc = -22 [ 1361.538189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1361.584665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1361.656847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1361.726802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1361.786801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1361.851699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1361.900670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1361.964825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1362.031688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1362.065899] LNet: 64776:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1362.108147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1362.151406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1362.193760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1362.226433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1362.256301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1362.298490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1362.327868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1362.352539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1366.580328] LNet: 64854:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1366.589440] LNetError: 64854:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1366.601330] LNet: Removed LNI 192.168.204.60@tcp [ 1366.618732] LNet: Removed LNI 192.168.204.60@tcp1 [ 1367.099516] Key type .llcrypt unregistered [ 1367.101644] Key type ._llcrypt unregistered [ 1387.383714] Key type ._llcrypt registered [ 1387.396620] Key type .llcrypt registered [ 1388.834744] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 15:41:01 (1788637261) [ 1390.853281] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 1392.881449] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 15:41:05 (1788637265) [ 1394.175369] Key type .llcrypt unregistered [ 1394.179515] Key type ._llcrypt unregistered [ 1402.946550] Key type ._llcrypt registered [ 1402.950883] Key type .llcrypt registered [ 1403.205439] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1403.219304] alg: No test for adler32 (adler32-zlib) [ 1404.216820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1404.247610] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 1415.617303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.160@tcp [ 1416.612138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.60@tcp -d 192.168.204.160@tcp -m GET -e network_timeout -r 1 [ 1416.647856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.60@tcp -d 192.168.204.60@tcp -m GET -e network_timeout -r 1 [ 1416.675958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.160@tcp [ 1416.690748] LNet: There was an unexpected network error while writing to 192.168.204.160: rc = -22 [ 1416.905364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1416.943523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1416.976275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1417.012840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1417.060706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1417.116819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1417.168655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1417.241506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1417.286156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1417.303485] LNet: 66397:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1417.329755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1417.372472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1417.417173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1417.477619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1417.539512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1417.568900] LNet: 1 local NIs in recovery (showing 1): 192.168.204.60@tcp [ 1417.576561] LNet: There was an unexpected network error while writing to 192.168.204.60: rc = -22 [ 1417.593049] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1417.846662] LNet: 66436:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1417.858035] LNetError: 66436:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1418.918906] LNet: Removed LNI 192.168.204.60@tcp [ 1419.518405] Key type .llcrypt unregistered [ 1419.519964] Key type ._llcrypt unregistered [ 1440.298331] Key type ._llcrypt registered [ 1440.300637] Key type .llcrypt registered [ 1440.443793] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1440.462447] alg: No test for adler32 (adler32-zlib) [ 1441.344938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1441.368867] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 1449.205693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.160@tcp [ 1458.962614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1459.001696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1459.031130] LNet: Added LNI 192.168.204.60@tcp1 [8/256/0/180] [ 1460.661554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.60@tcp -d 192.168.204.160@tcp -m GET -e network_timeout -r 1 [ 1460.711196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.60@tcp -d 192.168.204.160@tcp1 -m GET -e network_timeout -r 1 [ 1460.749499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.60@tcp -d 192.168.204.60@tcp -m GET -e network_timeout -r 1 [ 1460.792503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.60@tcp -d 192.168.204.60@tcp1 -m GET -e network_timeout -r 1 [ 1460.871625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.60@tcp1 -d 192.168.204.160@tcp -m GET -e network_timeout -r 1 [ 1460.954370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.60@tcp1 -d 192.168.204.160@tcp1 -m GET -e network_timeout -r 1 [ 1461.014764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.60@tcp1 -d 192.168.204.60@tcp -m GET -e network_timeout -r 1 [ 1461.087702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.60@tcp1 -d 192.168.204.60@tcp1 -m GET -e network_timeout -r 1 [ 1461.222581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.160@tcp [ 1461.270267] LNet: There was an unexpected network error while writing to 192.168.204.160: rc = -22 [ 1461.585913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1461.635314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1461.680720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1461.722440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1461.763073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1461.800429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1461.841441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1461.855553] LNet: 1 local NIs in recovery (showing 1): 192.168.204.60@tcp [ 1461.869466] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.160@tcp [ 1461.899507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1461.937804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1461.970616] LNet: 67313:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1462.005094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1462.074145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1462.119362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1462.161212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1462.218711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1462.345336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1466.822348] LNet: There was an unexpected network error while writing to 192.168.204.160: rc = -22 [ 1467.141689] LNet: 67387:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1467.150663] LNetError: 67387:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1467.158284] LNet: Removed LNI 192.168.204.60@tcp [ 1467.168334] LNet: Removed LNI 192.168.204.60@tcp1 [ 1467.577502] Key type .llcrypt unregistered [ 1467.581072] Key type ._llcrypt unregistered [ 1490.360951] Key type ._llcrypt registered [ 1490.363964] Key type .llcrypt registered [ 1491.814523] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 15:42:44 (1788637364) [ 1492.168635] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1492.179994] alg: No test for adler32 (adler32-zlib) [ 1493.110651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1493.233112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1493.300465] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 1494.168347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1494.190977] LNet: Added LNI 192.168.204.60@tcp1 [8/256/0/180] [ 1495.182046] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.60@tcp [ 1495.278425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1495.363443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1495.410798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1495.478839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1495.551371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1495.603854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.204.60@tcp [ 1495.636670] LNetError: 68486:0:(api-ni.c:4478:lnet_ni_set_healthv()) manually adding local NI 192.168.204.60@tcp to recovery [ 1496.223513] LNet: 1 local NIs in recovery (showing 1): 192.168.204.60@tcp [ 1496.223727] LNet: There was an unexpected network error while writing to 192.168.204.60: rc = -22 [ 1498.272571] LNet: There was an unexpected network error while writing to 192.168.204.60: rc = -22 [ 1499.185615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1502.370490] LNet: There was an unexpected network error while writing to 192.168.204.60: rc = -22 [ 1502.839679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1502.892637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1502.980992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1503.029419] LNet: 68593:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1503.035699] LNetError: 68593:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1504.096526] LNet: Removed LNI 192.168.204.60@tcp [ 1504.099734] LNetError: 68593:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1504.121762] LNet: Removed LNI 192.168.204.60@tcp1 [ 1504.186836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1504.335905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1504.375128] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 1505.075860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1505.091909] LNet: Added LNI 192.168.204.60@tcp1 [8/256/0/180] [ 1505.989987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.60@tcp [ 1506.023483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1506.067802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1506.112130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1506.136294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1506.165621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1506.193567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1506.216751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.204.60@tcp [ 1506.231349] LNetError: 68866:0:(api-ni.c:4478:lnet_ni_set_healthv()) manually adding local NI 192.168.204.60@tcp to recovery [ 1506.275241] LNet: There was an unexpected network error while writing to 192.168.204.60: rc = -22 [ 1508.665690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1512.421074] LNet: There was an unexpected network error while writing to 192.168.204.60: rc = -22 [ 1512.431768] LNet: Skipped 1 previous similar message [ 1516.921701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1517.031377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1517.069460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1525.303312] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 15:43:18 (1788637398) [ 1525.633779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1525.651910] LNet: 69573:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1525.657557] LNetError: 69573:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1525.663676] LNet: Removed LNI 192.168.204.60@tcp [ 1525.677540] LNet: Removed LNI 192.168.204.60@tcp1 [ 1525.709412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1525.770417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1525.799468] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 1526.712917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1526.730963] LNet: Added LNI 192.168.204.60@tcp1 [8/256/0/180] [ 1527.729992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.60@tcp [ 1527.802313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1527.838172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1527.902098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1527.949745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1527.991307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1528.030649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.204.60@tcp --health 0 [ 1528.136585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1528.799091] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.60@tcp [ 1529.826792] LNet: There was an unexpected network error while writing to 192.168.204.60: rc = -22 [ 1529.836759] LNet: Skipped 1 previous similar message [ 1543.952858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1544.144863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1544.213347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1544.241442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.204.60@tcp --health 500 [ 1544.363029] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1551.138304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1551.258852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1551.297029] LNet: 70066:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1551.306371] LNetError: 70066:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1551.317154] LNetError: 70066:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1551.327715] LNet: Removed LNI 192.168.204.60@tcp [ 1551.339986] LNet: Removed LNI 192.168.204.60@tcp1 [ 1551.399717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1551.525742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1551.571569] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 1552.436146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1552.471392] LNet: Added LNI 192.168.204.60@tcp1 [8/256/0/180] [ 1553.693991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.60@tcp [ 1553.730807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1553.810437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1553.857416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1553.908850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1553.947734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1553.992735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1554.025587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.204.60@tcp --health 0 [ 1555.554902] LNet: There was an unexpected network error while writing to 192.168.204.60: rc = -22 [ 1555.560023] LNet: Skipped 3 previous similar messages [ 1556.583415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1567.143622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1567.192989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1567.256507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1575.701510] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 15:44:08 (1788637448) [ 1576.301635] LNet: 71123:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1576.311675] LNetError: 71123:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1576.317032] LNetError: 71123:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1577.375865] LNet: Removed LNI 192.168.204.60@tcp [ 1577.399370] LNet: Removed LNI 192.168.204.60@tcp1 [ 1578.321145] Key type .llcrypt unregistered [ 1578.322981] Key type ._llcrypt unregistered [ 1585.695038] Key type ._llcrypt registered [ 1585.697777] Key type .llcrypt registered [ 1585.926265] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1585.943612] alg: No test for adler32 (adler32-zlib) [ 1586.847615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1586.871580] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 1598.189433] Lustre: DEBUG MARKER: Initial discovery [ 1598.244774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.204.160@tcp [ 1601.177941] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1601.253877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1601.313641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.204.160@tcp [ 1601.339117] LNet: There was an unexpected network error while writing to 192.168.204.160: rc = -22 [ 1601.380597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1601.462767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.204.160@tcp -r 1 -m GET -l 3 [ 1601.547661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.204.160@tcp -r 1 -m GET -e local_error [ 1601.593565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.204.160@tcp -r 1 -m PUT -l 6 [ 1602.272478] LNet: 1 local NIs in recovery (showing 1): 192.168.204.60@tcp [ 1603.227289] Lustre: DEBUG MARKER: Force oleg460-client.virtnet to discover 192.168.204.160@tcp (in background) [ 1603.486849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.204.160@tcp [ 1604.776922] Lustre: DEBUG MARKER: Force oleg460-server to discover 192.168.204.60@tcp [ 1606.623496] LNet: There was an unexpected network error while writing to 192.168.204.160: rc = -22 [ 1607.759632] Lustre: DEBUG MARKER: Wait for 71884 [ 1609.251962] Lustre: DEBUG MARKER: Finished wait on 71884 [ 1609.692433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay del -a [ 1609.760026] LNet: There was an unexpected network error while writing to 192.168.204.160: rc = -22 [ 1609.788028] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1610.124454] LNet: 72333:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1610.132740] LNetError: 72333:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1610.160932] LNet: Removed LNI 192.168.204.60@tcp [ 1610.562467] Key type .llcrypt unregistered [ 1610.564950] Key type ._llcrypt unregistered [ 1629.591555] Key type ._llcrypt registered [ 1629.593785] Key type .llcrypt registered [ 1631.044299] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 15:45:03 (1788637503) [ 1631.486281] Key type .llcrypt unregistered [ 1631.490443] Key type ._llcrypt unregistered [ 1637.111097] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1637.266768] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1637.271407] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1637.289154] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1639.837407] Key type ._llcrypt registered [ 1639.840545] Key type .llcrypt registered [ 1639.965264] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1639.977181] alg: No test for adler32 (adler32-zlib) [ 1640.959727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1641.050896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1641.084077] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 1641.826129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1641.848718] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1650.274178] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 15:45:23 (1788637523) [ 1650.371962] LNet: 74398:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1650.397188] LNetError: 74398:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1650.408176] LNet: Removed LNI 10.1.2.3@tcp [ 1650.445510] LNet: Removed LNI 192.168.204.60@tcp [ 1651.166438] Key type .llcrypt unregistered [ 1651.168434] Key type ._llcrypt unregistered [ 1657.953217] Key type ._llcrypt registered [ 1657.955590] Key type .llcrypt registered [ 1658.119155] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1658.143621] alg: No test for adler32 (adler32-zlib) [ 1659.100805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1659.149448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1659.174202] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 1659.794324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1659.814616] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1660.772505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.204.60@tcp [ 1661.855844] LNetError: Unexpected error -22 connecting to 192.168.204.60@tcp at host 192.168.204.60:988 [ 1667.344194] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 15:45:40 (1788637540) [ 1667.681485] LNet: 75536:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1667.688363] LNetError: 75536:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1667.693519] LNet: Removed LNI 10.1.2.3@tcp [ 1668.716394] LNet: Removed LNI 192.168.204.60@tcp [ 1669.419421] Key type .llcrypt unregistered [ 1669.422206] Key type ._llcrypt unregistered [ 1676.885805] Key type ._llcrypt registered [ 1676.889254] Key type .llcrypt registered [ 1677.207184] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1677.232448] alg: No test for adler32 (adler32-zlib) [ 1678.223718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1678.295316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1678.323789] LNet: Added LNI 192.168.204.60@tcp1 [8/256/0/180] [ 1679.073977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1679.108066] LNet: Added LNI 192.168.204.60@tcp2 [8/256/0/180] [ 1680.191539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.204.60@tcp1 --nid 192.168.204.60@tcp2 [ 1684.242337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.60@tcp1 192.168.204.60@tcp2 [ 1684.275692] LNetError: 76368:0:(lib-move.c:2094:lnet_handle_find_routed_path()) Unable to send message from 192.168.204.60@tcp1 to 192.168.204.60@tcp2 - Route table may be misconfigured [ 1684.295333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.60@tcp2 192.168.204.60@tcp1 [ 1690.505527] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 15:46:03 (1788637563) [ 1690.654821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1690.701529] LNet: 76935:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1690.712955] LNetError: 76935:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1690.722397] LNet: Removed LNI 192.168.204.60@tcp1 [ 1691.754458] LNet: Removed LNI 192.168.204.60@tcp2 [ 1691.867403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1691.971040] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1692.048778] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 1693.406641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1693.437289] LNet: Added LNI 192.168.204.60@tcp1 [8/256/0/180] [ 1694.689948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.60@tcp [ 1694.748088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.60@tcp 192.168.204.60@tcp [ 1694.801043] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.60@tcp1 192.168.204.60@tcp1 [ 1694.879815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.204.60@tcp -d 192.168.204.60@tcp -e network_timeout [ 1694.945584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.204.60@tcp -d 192.168.204.60@tcp1 -e network_timeout [ 1695.026263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.204.60@tcp1 -d 192.168.204.60@tcp -e network_timeout [ 1695.082963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.204.60@tcp1 -d 192.168.204.60@tcp1 -e network_timeout [ 1695.126462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.60@tcp [ 1695.167152] LNet: There was an unexpected network error while writing to 192.168.204.60: rc = -22 [ 1695.205823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1695.436700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1695.651648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1696.033182] LNet: 1 local NIs in recovery (showing 1): 192.168.204.60@tcp [ 1703.085549] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 15:46:15 (1788637575) [ 1703.280666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1703.323373] LNet: 77803:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1703.339517] LNetError: 77803:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1703.344500] LNetError: 77803:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1704.415559] LNet: Removed LNI 192.168.204.60@tcp [ 1704.422765] LNetError: 77803:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1704.442242] LNet: Removed LNI 192.168.204.60@tcp1 [ 1704.506890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1704.660604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1705.060705] LNet: 77840:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1705.410645] Key type .llcrypt unregistered [ 1705.413231] Key type ._llcrypt unregistered [ 1718.726813] Key type ._llcrypt registered [ 1718.729865] Key type .llcrypt registered [ 1720.273987] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 15:46:32 (1788637592) [ 1720.711382] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1720.739599] alg: No test for adler32 (adler32-zlib) [ 1721.725983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1723.806591] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1726.184686] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 15:46:38 (1788637598) [ 1726.347138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1726.376388] LNet: 78933:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1726.413874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1726.474706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1726.512904] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 1727.707985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1727.739634] LNet: Added LNI 192.168.204.60@tcp1 [8/256/0/180] [ 1729.035642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.60@tcp [ 1729.104926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.60@tcp1 [ 1729.137682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.60@tcp1 [ 1736.864860] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 15:46:49 (1788637609) [ 1739.049506] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg460-server [ 1741.152367] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 15:46:53 (1788637613) [ 1743.325377] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg460-server [ 1745.257801] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 15:46:58 (1788637618) [ 1747.606569] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg460-server [ 1750.266218] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 15:47:02 (1788637622) [ 1752.101819] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg460-server [ 1754.525993] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 15:47:07 (1788637627) [ 1757.164605] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg460-server [ 1759.052456] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 15:47:11 (1788637631) [ 1761.241051] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg460-server [ 1763.731674] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 15:47:16 (1788637636) [ 1766.225569] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg460-server [ 1768.466616] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 15:47:21 (1788637641) [ 1771.355282] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg460-server [ 1773.505984] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 15:47:26 (1788637646) [ 1775.702547] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg460-server [ 1777.967482] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 15:47:30 (1788637650) [ 1778.084354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1778.121210] LNet: 82808:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1778.134655] LNetError: 82808:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1778.143584] LNet: Removed LNI 192.168.204.60@tcp [ 1779.181438] LNet: Removed LNI 192.168.204.60@tcp1 [ 1779.252926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1779.344160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1779.409364] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 1780.136741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1780.232040] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.60@tcp [ 1780.368805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1780.385816] LNet: 82963:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1780.390301] LNetError: 82963:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1780.396556] LNetError: 82963:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1780.407382] LNet: Removed LNI 192.168.204.60@tcp [ 1780.423027] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1780.468907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1780.494625] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 1781.131728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.60@tcp [ 1781.373257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1781.413804] LNetError: 83108:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1782.507893] LNet: Removed LNI 192.168.204.60@tcp [ 1782.598369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1782.668233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1782.745360] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 1783.634697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1783.740748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.60@tcp [ 1785.106746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1785.150539] LNet: 83265:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1785.158898] LNet: 83265:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1785.171706] LNetError: 83265:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1785.197931] LNet: Removed LNI 192.168.204.60@tcp [ 1785.223891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1785.288765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1785.312444] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 1786.479127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.60@tcp [ 1786.809636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1787.891369] LNet: Removed LNI 192.168.204.60@tcp [ 1787.929787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1788.035208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1788.073744] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 1789.109862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1789.188745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.60@tcp [ 1789.495934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1789.533530] LNet: 83558:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1789.540698] LNet: 83558:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1789.548983] LNetError: 83558:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1789.555893] LNetError: 83558:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1790.643348] LNet: Removed LNI 192.168.204.60@tcp [ 1790.685411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1790.741369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1790.772794] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 1791.600778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.60@tcp [ 1791.938638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1793.003602] LNet: Removed LNI 192.168.204.60@tcp [ 1793.118792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1793.233041] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1793.258011] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 1794.519825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1794.650276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.60@tcp [ 1795.051942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1796.140707] LNet: Removed LNI 192.168.204.60@tcp [ 1796.211685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1796.310682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1796.351463] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 1797.364069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.60@tcp [ 1798.878254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1798.906394] LNet: 84005:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1798.914477] LNet: 84005:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 1798.932474] LNetError: 84005:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1798.940118] LNetError: 84005:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1798.957310] LNet: Removed LNI 192.168.204.60@tcp [ 1799.009814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1799.071688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1799.094589] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 1799.994790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1800.078378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.60@tcp [ 1800.384705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1801.449309] LNet: Removed LNI 192.168.204.60@tcp [ 1801.502504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1801.609770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1801.641967] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 1802.504527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.60@tcp [ 1802.862771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1803.943137] LNet: Removed LNI 192.168.204.60@tcp [ 1804.007000] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1804.097813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1804.125903] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 1804.993699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1805.113510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.60@tcp [ 1805.448795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1806.507946] LNet: Removed LNI 192.168.204.60@tcp [ 1806.543052] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1806.603869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1806.637542] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 1807.618716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.60@tcp [ 1807.872530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1808.936054] LNet: Removed LNI 192.168.204.60@tcp [ 1808.973634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1809.039246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1809.076798] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 1809.833759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1809.893509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1809.958054] LNet: Removed LNI 192.168.204.60@tcp [ 1809.989878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1810.078447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1810.125241] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 1811.016788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1811.097877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1811.139393] LNet: Removed LNI 192.168.204.60@tcp [ 1811.177611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1811.268521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 128 [ 1811.393546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1811.426341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1811.495829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 1000000000000000000000000 [ 1811.594591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1811.621701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1811.666985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer foo [ 1811.765424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1811.825339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1811.922657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 8 [ 1811.984534] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 1812.837940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1812.886947] LNet: Removed LNI 192.168.204.60@tcp [ 1812.917274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1812.995878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1819.716492] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 15:48:12 (1788637692) [ 1819.770816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1819.792689] LNet: 85607:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1819.797938] LNet: 85607:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 10 previous similar messages [ 1819.815430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1819.857821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1819.878065] LNet: Added LNI 192.168.204.60@tcp231 [8/256/0/180] [ 1820.916592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1820.953513] LNetError: 85751:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1820.961220] LNetError: 85751:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 1820.983217] LNet: Removed LNI 192.168.204.60@tcp231 [ 1821.007080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1821.114161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1821.155446] LNet: Added LNI 192.168.204.60@tcp231 [8/256/0/180] [ 1821.161035] LNet: 85759:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1821.684755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1821.742919] LNet: Removed LNI 192.168.204.60@tcp231 [ 1821.761697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1821.790890] LNet: Added LNI 192.168.204.60@tcp231 [8/256/0/180] [ 1822.884690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1822.930850] LNet: Removed LNI 192.168.204.60@tcp231 [ 1822.957426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1823.059845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1823.085800] LNet: Added LNI 192.168.204.60@tcp231 [8/256/0/0] [ 1823.089183] LNet: 85911:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1823.553321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1823.601150] LNet: Removed LNI 192.168.204.60@tcp231 [ 1823.621827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1823.654816] LNet: Added LNI 192.168.204.60@tcp231 [8/256/0/0] [ 1824.901597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1824.950581] LNet: Removed LNI 192.168.204.60@tcp231 [ 1824.989916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1825.078594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1825.114796] LNet: Added LNI 192.168.204.60@tcp231 [8/256/0/60] [ 1825.122704] LNet: 86062:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1825.611424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1825.649459] LNet: Removed LNI 192.168.204.60@tcp231 [ 1825.677114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1825.720129] LNet: Added LNI 192.168.204.60@tcp231 [8/256/0/60] [ 1827.091715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1827.141596] LNet: Removed LNI 192.168.204.60@tcp231 [ 1827.160729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1827.250496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1827.308708] LNet: Added LNI 192.168.204.60@tcp231 [8/256/0/180] [ 1827.321458] LNet: 86214:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1827.926261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1827.960130] LNet: Removed LNI 192.168.204.60@tcp231 [ 1827.989509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1828.025264] LNet: Added LNI 192.168.204.60@tcp231 [8/256/0/180] [ 1836.030156] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 15:48:29 (1788637709) [ 1836.292610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1836.318829] LNet: Removed LNI 192.168.204.60@tcp231 [ 1836.339884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1836.425995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1836.455322] LNet: Added LNI 192.168.204.60@tcp232 [8/256/0/180] [ 1837.105975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1837.152322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1837.188978] LNet: Removed LNI 192.168.204.60@tcp232 [ 1837.199683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1837.224117] LNet: Added LNI 192.168.204.60@tcp232 [8/256/0/180] [ 1837.902556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1837.936498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1837.962200] LNet: Removed LNI 192.168.204.60@tcp232 [ 1837.975800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1838.000382] LNet: Added LNI 192.168.204.60@tcp232 [8/256/0/180] [ 1838.673611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1838.715992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1838.760228] LNet: Removed LNI 192.168.204.60@tcp232 [ 1838.777878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1838.811887] LNet: Added LNI 192.168.204.60@tcp232 [8/256/0/180] [ 1839.623726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1839.675537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1839.739211] LNet: Removed LNI 192.168.204.60@tcp232 [ 1839.759259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1846.565375] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 15:48:39 (1788637719) [ 1848.075271] Key type .llcrypt unregistered [ 1848.078266] Key type ._llcrypt unregistered [ 1854.774285] Key type ._llcrypt registered [ 1854.780371] Key type .llcrypt registered [ 1855.049479] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1855.062829] alg: No test for adler32 (adler32-zlib) [ 1855.983912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1856.019535] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 1864.780545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.160@tcp [ 1874.993785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1875.060713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1875.090788] LNet: Added LNI 192.168.204.60@tcp1 [8/256/0/180] [ 1876.567594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.60@tcp -d 192.168.204.160@tcp -m GET -e local_interrupt -r 2 [ 1876.621522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.60@tcp -d 192.168.204.160@tcp1 -m GET -e local_interrupt -r 2 [ 1876.671965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.60@tcp -d 192.168.204.60@tcp -m GET -e local_interrupt -r 2 [ 1876.706926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.60@tcp -d 192.168.204.60@tcp1 -m GET -e local_interrupt -r 2 [ 1876.743395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.60@tcp1 -d 192.168.204.160@tcp -m GET -e local_interrupt -r 2 [ 1876.791610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.60@tcp1 -d 192.168.204.160@tcp1 -m GET -e local_interrupt -r 2 [ 1876.827435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.60@tcp1 -d 192.168.204.60@tcp -m GET -e local_interrupt -r 2 [ 1876.864183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.60@tcp1 -d 192.168.204.60@tcp1 -m GET -e local_interrupt -r 2 [ 1876.902389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.60@tcp 192.168.204.160@tcp [ 1876.919605] LNet: There was an unexpected network error while writing to 192.168.204.160: rc = -22 [ 1876.924560] LNet: 1 local NIs in recovery (showing 1): 192.168.204.60@tcp [ 1877.170365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1877.212482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1877.240325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1877.273854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1877.302559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1877.327375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1877.360590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1877.394753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1877.440470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1877.457692] LNet: 88716:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1877.478946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1877.512084] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1877.543569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1877.581330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1877.618346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1877.656342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1878.129957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.60@tcp -d 192.168.204.160@tcp -m GET -e local_dropped -r 2 [ 1878.163377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.60@tcp -d 192.168.204.160@tcp1 -m GET -e local_dropped -r 2 [ 1878.199901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.60@tcp -d 192.168.204.60@tcp -m GET -e local_dropped -r 2 [ 1878.248344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.60@tcp -d 192.168.204.60@tcp1 -m GET -e local_dropped -r 2 [ 1878.289168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.60@tcp1 -d 192.168.204.160@tcp -m GET -e local_dropped -r 2 [ 1878.319491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.60@tcp1 -d 192.168.204.160@tcp1 -m GET -e local_dropped -r 2 [ 1878.354360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.60@tcp1 -d 192.168.204.60@tcp -m GET -e local_dropped -r 2 [ 1878.388831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.60@tcp1 -d 192.168.204.60@tcp1 -m GET -e local_dropped -r 2 [ 1878.435297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.60@tcp 192.168.204.160@tcp [ 1878.462851] LNet: There was an unexpected network error while writing to 192.168.204.160: rc = -22 [ 1878.729891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1878.764668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1878.821867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1878.867683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1878.913504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1878.968848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1879.035382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1879.078077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1879.110845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1879.138459] LNet: 88839:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1879.170593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1879.237705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1879.287578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1879.337443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1879.390126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1879.422149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1879.930336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.60@tcp -d 192.168.204.160@tcp -m GET -e local_aborted -r 2 [ 1879.962851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.60@tcp -d 192.168.204.160@tcp1 -m GET -e local_aborted -r 2 [ 1879.998124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.60@tcp -d 192.168.204.60@tcp -m GET -e local_aborted -r 2 [ 1880.033945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.60@tcp -d 192.168.204.60@tcp1 -m GET -e local_aborted -r 2 [ 1880.112841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.60@tcp1 -d 192.168.204.160@tcp -m GET -e local_aborted -r 2 [ 1880.175926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.60@tcp1 -d 192.168.204.160@tcp1 -m GET -e local_aborted -r 2 [ 1880.265049] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.60@tcp1 -d 192.168.204.60@tcp -m GET -e local_aborted -r 2 [ 1880.326665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.60@tcp1 -d 192.168.204.60@tcp1 -m GET -e local_aborted -r 2 [ 1880.376643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.60@tcp 192.168.204.160@tcp [ 1880.410726] LNet: There was an unexpected network error while writing to 192.168.204.160: rc = -22 [ 1880.687793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1880.720078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1880.762264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1880.804120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1880.832983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1880.888698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1880.926377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1880.963114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1881.015738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1881.034239] LNet: 88961:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1881.061887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1881.105035] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1881.177541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1881.224369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1881.250973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1881.300792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1881.831263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.60@tcp -d 192.168.204.160@tcp -m GET -e local_no_route -r 2 [ 1881.881476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.60@tcp -d 192.168.204.160@tcp1 -m GET -e local_no_route -r 2 [ 1881.955268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.60@tcp -d 192.168.204.60@tcp -m GET -e local_no_route -r 2 [ 1882.008919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.60@tcp -d 192.168.204.60@tcp1 -m GET -e local_no_route -r 2 [ 1882.057527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.60@tcp1 -d 192.168.204.160@tcp -m GET -e local_no_route -r 2 [ 1882.090781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.60@tcp1 -d 192.168.204.160@tcp1 -m GET -e local_no_route -r 2 [ 1882.146234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.60@tcp1 -d 192.168.204.60@tcp -m GET -e local_no_route -r 2 [ 1882.185898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.60@tcp1 -d 192.168.204.60@tcp1 -m GET -e local_no_route -r 2 [ 1882.228768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.60@tcp 192.168.204.160@tcp [ 1882.540023] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1882.570593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1882.604776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1882.668881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1882.697537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1882.724324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1882.755913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1882.789327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1882.832914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1882.888189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1882.927546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1882.963283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1882.993951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1883.025298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1883.061581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1883.604082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.60@tcp -d 192.168.204.160@tcp -m GET -e local_timeout -r 2 [ 1883.642926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.60@tcp -d 192.168.204.160@tcp1 -m GET -e local_timeout -r 2 [ 1883.678928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.60@tcp -d 192.168.204.60@tcp -m GET -e local_timeout -r 2 [ 1883.716064] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.60@tcp -d 192.168.204.60@tcp1 -m GET -e local_timeout -r 2 [ 1883.756757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.60@tcp1 -d 192.168.204.160@tcp -m GET -e local_timeout -r 2 [ 1883.801710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.60@tcp1 -d 192.168.204.160@tcp1 -m GET -e local_timeout -r 2 [ 1883.843595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.60@tcp1 -d 192.168.204.60@tcp -m GET -e local_timeout -r 2 [ 1883.877746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.60@tcp1 -d 192.168.204.60@tcp1 -m GET -e local_timeout -r 2 [ 1883.912073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.60@tcp 192.168.204.160@tcp [ 1883.931763] LNet: There was an unexpected network error while writing to 192.168.204.160: rc = -22 [ 1883.939330] LNet: Skipped 1 previous similar message [ 1884.145070] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1884.173826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1884.196855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1884.231951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1884.270789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1884.313771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1884.341124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1884.368720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1884.400580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1884.413872] LNet: 89206:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1884.419700] LNet: 89206:0:(api-ni.c:301:recovery_interval_set()) Skipped 1 previous similar message [ 1884.434752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1884.469846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1884.497650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1884.528154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1884.557613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1884.585494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1885.235266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.60@tcp -d 192.168.204.160@tcp -m GET -e remote_dropped -r 2 [ 1885.276455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.60@tcp -d 192.168.204.160@tcp1 -m GET -e remote_dropped -r 2 [ 1885.325553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.60@tcp -d 192.168.204.60@tcp -m GET -e remote_dropped -r 2 [ 1885.370568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.60@tcp -d 192.168.204.60@tcp1 -m GET -e remote_dropped -r 2 [ 1885.411878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.60@tcp1 -d 192.168.204.160@tcp -m GET -e remote_dropped -r 2 [ 1885.456865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.60@tcp1 -d 192.168.204.160@tcp1 -m GET -e remote_dropped -r 2 [ 1885.509813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.60@tcp1 -d 192.168.204.60@tcp -m GET -e remote_dropped -r 2 [ 1885.557821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.60@tcp1 -d 192.168.204.60@tcp1 -m GET -e remote_dropped -r 2 [ 1885.611623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.60@tcp 192.168.204.160@tcp [ 1885.633762] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.160@tcp [ 1885.840749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1885.886386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1885.918640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1885.962914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1886.003675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1886.055877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1886.097618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1886.155269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1886.197465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1886.223883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1886.247912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1886.274831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1886.303652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1886.339871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1886.373641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1892.311935] LNet: 89402:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1892.322432] LNetError: 89402:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1892.332143] LNet: Removed LNI 192.168.204.60@tcp [ 1892.355271] LNet: Removed LNI 192.168.204.60@tcp1 [ 1893.027578] Key type .llcrypt unregistered [ 1893.032076] Key type ._llcrypt unregistered [ 1911.028530] Key type ._llcrypt registered [ 1911.035659] Key type .llcrypt registered [ 1912.648326] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 15:49:45 (1788637785) [ 1913.039854] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1913.051890] alg: No test for adler32 (adler32-zlib) [ 1913.905749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1913.948873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1913.963619] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 1914.807926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1914.859227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.204.60@tcp [ 1914.896757] LNetError: 90379:0:(api-ni.c:4478:lnet_ni_set_healthv()) manually adding local NI 192.168.204.60@tcp to recovery [ 1914.977125] LNet: 1 local NIs in recovery (showing 1): 192.168.204.60@tcp [ 1915.022763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.204.60@tcp [ 1915.149638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1922.665242] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 15:49:55 (1788637795) [ 1922.769434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1922.794710] LNet: 90962:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1922.800495] LNetError: 90962:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1922.818630] LNet: Removed LNI 192.168.204.60@tcp [ 1922.872953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1923.049458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1923.125812] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 1923.939981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1923.963209] LNet: Added LNI 192.168.204.60@tcp1 [8/256/0/180] [ 1924.897447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.204.60@tcp [ 1924.911828] LNetError: 91236:0:(api-ni.c:4478:lnet_ni_set_healthv()) manually adding local NI 192.168.204.60@tcp to recovery [ 1925.032406] LNet: There was an unexpected network error while writing to 192.168.204.60: rc = -22 [ 1926.150192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.204.60@tcp [ 1933.523543] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 15:50:06 (1788637806) [ 1935.645551] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg460-server [ 1937.948584] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 15:50:10 (1788637810) [ 1940.020441] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg460-server [ 1942.021208] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 15:50:14 (1788637814) [ 1944.246138] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg460-server [ 1946.560547] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 15:50:19 (1788637819) [ 1946.754864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1946.791693] LNet: 92858:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1946.801484] LNetError: 92858:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1946.813424] LNet: Removed LNI 192.168.204.60@tcp [ 1946.839830] LNet: Removed LNI 192.168.204.60@tcp1 [ 1946.865548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1946.936569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1946.970722] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 1948.046682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1948.120250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1948.184766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 1948.337121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1948.391749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 1948.541677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1957.497377] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 15:50:29 (1788637829) [ 1957.649501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1957.703603] LNet: 93582:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1957.707477] LNetError: 93582:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1957.714506] LNetError: 93582:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1957.743413] LNet: Removed LNI 192.168.204.60@tcp [ 1957.776394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1957.873300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1957.904323] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 1965.733399] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 15:50:38 (1788637838) [ 1965.914980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1965.995865] LNet: 94291:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1966.013370] LNetError: 94291:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1966.066045] LNet: Removed LNI 192.168.204.60@tcp [ 1966.113956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1966.220636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1966.300194] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 1973.696113] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 15:50:46 (1788637846) [ 1975.051330] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 1976.778099] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 15:50:49 (1788637849) [ 1977.735641] LNet: 95329:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1977.740924] LNetError: 95329:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1977.753320] LNet: Removed LNI 192.168.204.60@tcp [ 1978.339843] Key type .llcrypt unregistered [ 1978.342801] Key type ._llcrypt unregistered [ 1985.141660] Key type ._llcrypt registered [ 1985.147272] Key type .llcrypt registered [ 1985.399312] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1985.415899] alg: No test for adler32 (adler32-zlib) [ 1986.388933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1986.418265] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 1993.873679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.160@tcp [ 1999.017884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.204.160@tcp [ 1999.426733] LNet: 95745:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1999.438725] LNetError: 95745:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1999.455393] LNet: Removed LNI 192.168.204.60@tcp [ 1999.951690] Key type .llcrypt unregistered [ 1999.957077] Key type ._llcrypt unregistered [ 2010.592906] Key type ._llcrypt registered [ 2010.600409] Key type .llcrypt registered [ 2012.004681] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 2013.934671] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 2015.183705] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 15:51:28 (1788637888) [ 2016.758325] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 [ 2018.165332] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 15:51:31 (1788637891) [ 2020.009271] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg460-server [ 2021.482220] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 15:51:34 (1788637894) [ 2022.929344] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg460-server [ 2024.402462] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 15:51:37 (1788637897) [ 2026.326635] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2026.445892] alg: No test for adler32 (adler32-zlib) [ 2027.620752] Lustre: Lustre: Build Version: 2.17.58_2_g6837bc1 [ 2027.892310] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 2029.647289] Key type lgssc registered [ 2030.957457] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2051.747505] Key type lgssc unregistered [ 2052.080642] LNet: 98915:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2052.086083] LNetError: 98915:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2052.107237] LNet: Removed LNI 192.168.204.60@tcp [ 2053.080568] Key type .llcrypt unregistered [ 2053.086532] Key type ._llcrypt unregistered [ 2068.220742] Key type ._llcrypt registered [ 2068.225427] Key type .llcrypt registered [ 2069.335206] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 15:52:22 (1788637942) [ 2069.738269] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2069.774990] alg: No test for adler32 (adler32-zlib) [ 2070.693544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2070.811678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2070.854575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2070.924328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2070.967232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2077.712327] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 15:52:30 (1788637950) [ 2078.017210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2078.040857] LNet: 100328:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2078.065720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2078.138712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2078.172352] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 2079.062188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2079.085629] LNet: 100472:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2079.094665] LNetError: 100472:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2079.106868] LNet: Removed LNI 192.168.204.60@tcp [ 2079.124651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2079.192295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 2079.214697] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 2079.573419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2079.596432] LNet: Removed LNI 192.168.204.60@tcp [ 2079.625503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2079.685553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 2079.729958] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 2087.503425] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 15:52:40 (1788637960) [ 2087.727832] LNet: 101087:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2087.738031] LNet: 101087:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 2087.760048] LNetError: 101087:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2087.775909] LNetError: 101087:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2087.811394] LNet: Removed LNI 192.168.204.60@tcp [ 2088.542662] Key type .llcrypt unregistered [ 2088.545671] Key type ._llcrypt unregistered [ 2096.904247] Key type ._llcrypt registered [ 2096.906629] Key type .llcrypt registered [ 2097.245743] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2097.279560] alg: No test for adler32 (adler32-zlib) [ 2098.349848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2098.501330] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 2098.507184] LNet: *** cfs_fail_loc=e003, val=100*** [ 2099.715707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.60@tcp [ 2099.767250] LNet: *** cfs_fail_loc=e003, val=100*** [ 2099.770865] LNet: Skipped 2 previous similar messages [ 2099.788609] Ping reply received with mapping: [ 2099.791520] found metadata reply NI[192.168.204.60@tcp] [ 2099.796557] 3c cc a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 2099.804045] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 2099.805886] 5a 5a 5a 5a 5a 5a 5a 5a [ 2100.035952] LNet: 101485:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2100.050225] LNetError: 101485:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2100.076312] LNet: Removed LNI 192.168.204.60@tcp [ 2100.747843] Key type .llcrypt unregistered [ 2100.752976] Key type ._llcrypt unregistered [ 2112.864371] Key type ._llcrypt registered [ 2112.868307] Key type .llcrypt registered [ 2114.122231] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 15:53:06 (1788637986) [ 2115.097547] Key type .llcrypt unregistered [ 2115.103879] Key type ._llcrypt unregistered [ 2121.739692] Key type ._llcrypt registered [ 2121.742439] Key type .llcrypt registered [ 2122.013252] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2122.049259] alg: No test for adler32 (adler32-zlib) [ 2123.176535] LNetError: 102521:0:(api-ni.c:2855:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 2123.526425] Key type .llcrypt unregistered [ 2123.528700] Key type ._llcrypt unregistered [ 2128.779874] Key type ._llcrypt registered [ 2128.783697] Key type .llcrypt registered [ 2129.986265] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 15:53:22 (1788638002) [ 2130.680290] Key type .llcrypt unregistered [ 2130.683138] Key type ._llcrypt unregistered [ 2137.158137] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2137.163413] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2139.645234] Key type ._llcrypt registered [ 2139.647302] Key type .llcrypt registered [ 2139.911218] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2139.934245] alg: No test for adler32 (adler32-zlib) [ 2140.867064] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2140.929961] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2153.754822] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 15:53:46 (1788638026) [ 2153.942223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2154.014448] LNet: 104306:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2154.032643] LNetError: 104306:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2154.068941] LNet: Removed LNI 10.1.2.3@tcp [ 2154.155521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2155.941626] Key type .llcrypt unregistered [ 2155.957583] Key type ._llcrypt unregistered [ 2165.013184] Key type ._llcrypt registered [ 2165.023844] Key type .llcrypt registered [ 2165.220542] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2165.231848] alg: No test for adler32 (adler32-zlib) [ 2166.206989] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 2166.275915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.60@tcp [ 2166.502389] LNet: 104597:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2166.518722] LNetError: 104597:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2167.592222] LNet: Removed LNI 192.168.204.60@tcp [ 2168.384433] Key type .llcrypt unregistered [ 2168.388886] Key type ._llcrypt unregistered [ 2181.980215] Key type ._llcrypt registered [ 2181.990661] Key type .llcrypt registered [ 2183.878848] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 15:54:15 (1788638055) [ 2184.355515] Key type .llcrypt unregistered [ 2184.360206] Key type ._llcrypt unregistered [ 2191.805368] Key type ._llcrypt registered [ 2191.817583] Key type .llcrypt registered [ 2192.289768] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2192.324534] alg: No test for adler32 (adler32-zlib) [ 2203.507269] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 15:54:35 (1788638075) [ 2203.709823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2203.794145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2203.969593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2204.028859] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 2205.449117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2205.504452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 2205.551740] LNet: Removed LNI 192.168.204.60@tcp [ 2206.084893] LNet: 106336:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2206.740729] Key type .llcrypt unregistered [ 2206.747336] Key type ._llcrypt unregistered [ 2219.895198] Key type ._llcrypt registered [ 2219.897600] Key type .llcrypt registered [ 2221.270434] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 15:54:54 (1788638094) [ 2221.908343] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2221.933573] alg: No test for adler32 (adler32-zlib) [ 2223.005374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2223.096748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2223.124809] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 2224.079794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.60@tcp [ 2224.144391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.204.60@tcp [ 2224.176205] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 2224.181712] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 2224.191167] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2224, 0, 0 [ 2224.199368] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 2224.209139] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 2224.212467] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 2224.221579] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2224, 0, 0 [ 2224.225303] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 2224.227046] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2224.230204] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 2224.232967] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2224, 0, 2274 [ 2224.238134] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 2224.243996] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2224.247374] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 2224.253309] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2224, 0, 2274 [ 2224.258469] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 2231.650915] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 15:55:04 (1788638104) [ 2232.304913] LNet: 107873:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2232.314936] LNetError: 107873:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2232.335263] LNet: Removed LNI 192.168.204.60@tcp [ 2233.245774] Key type .llcrypt unregistered [ 2233.250300] Key type ._llcrypt unregistered [ 2241.575804] Key type ._llcrypt registered [ 2241.579761] Key type .llcrypt registered [ 2241.967839] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2241.992380] alg: No test for adler32 (adler32-zlib) [ 2242.979819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2243.029170] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 2252.024824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.160@tcp [ 2261.222994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2261.309501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2261.346203] LNet: Added LNI 192.168.204.60@tcp1 [8/256/0/180] [ 2262.384927] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2262.600724] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2262.606361] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2262.609180] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2264.972533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 2264.994550] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 2280.026184] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 15:55:52 (1788638152) [ 2280.318454] LNet: 109392:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2280.327547] LNetError: 109392:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2280.333876] LNet: Removed LNI 192.168.204.60@tcp [ 2281.375168] LNet: Removed LNI 192.168.204.60@tcp1 [ 2281.380520] LNetError: 109392:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2281.390722] LNetError: 109392:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2281.410434] LNet: Removed LNI 10.1.2.3@tcp99 [ 2282.368515] Key type .llcrypt unregistered [ 2282.370477] Key type ._llcrypt unregistered [ 2290.289801] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2290.294191] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2292.949425] Key type ._llcrypt registered [ 2292.951461] Key type .llcrypt registered [ 2293.179579] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2293.205514] alg: No test for adler32 (adler32-zlib) [ 2294.223473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2294.276991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2294.295899] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 2295.337637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2295.358893] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2297.654459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 2297.708532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2297.782104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 2297.815037] LNetError: 110135:0:(peer.c:2366:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 2297.869713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2297.926231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.60@tcp [ 2298.387486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2298.491274] LNet: 110155:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2298.500488] LNetError: 110155:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2298.510363] LNet: Removed LNI 10.1.2.3@tcp [ 2299.564360] LNet: Removed LNI 192.168.204.60@tcp [ 2300.340986] Key type .llcrypt unregistered [ 2300.359102] Key type ._llcrypt unregistered [ 2314.435662] Key type ._llcrypt registered [ 2314.440326] Key type .llcrypt registered [ 2316.714281] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 15:56:28 (1788638188) [ 2317.309729] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2317.335900] alg: No test for adler32 (adler32-zlib) [ 2318.322759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2318.412593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2318.446362] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 2319.724309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.60@tcp [ 2319.839384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg460-client@tcp [ 2327.385549] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 15:56:40 (1788638200) [ 2327.675783] LNet: 111687:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2327.689990] LNetError: 111687:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2328.741975] LNet: Removed LNI 192.168.204.60@tcp [ 2329.455538] Key type .llcrypt unregistered [ 2329.461192] Key type ._llcrypt unregistered [ 2336.918573] Key type ._llcrypt registered [ 2336.921349] Key type .llcrypt registered [ 2337.144949] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2337.166161] alg: No test for adler32 (adler32-zlib) [ 2338.171430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 2338.212339] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 2338.262043] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 2345.837489] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 15:56:58 (1788638218) [ 2347.795406] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 2350.021771] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 15:57:02 (1788638222) [ 2351.486852] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 2353.240827] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 15:57:06 (1788638226) [ 2353.592057] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2353.624825] LNet: 113108:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2353.636885] LNetError: 113108:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2353.663920] LNet: Removed LNI 192.168.204.60@tcp [ 2353.701409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2353.758417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2353.794940] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 2354.583928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2354.668881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2362.353906] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 15:57:15 (1788638235) [ 2362.460685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2362.500965] LNet: 113810:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2362.510073] LNetError: 113810:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2362.528490] LNet: Removed LNI 192.168.204.60@tcp [ 2362.586153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2362.704499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 2362.743394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2363.507531] Key type .llcrypt unregistered [ 2363.509682] Key type ._llcrypt unregistered [ 2375.802772] Key type ._llcrypt registered [ 2375.807932] Key type .llcrypt registered [ 2378.043976] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 15:57:29 (1788638249) [ 2378.589613] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2378.614758] alg: No test for adler32 (adler32-zlib) [ 2379.751329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2379.868768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2379.919494] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 2380.795484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2380.842348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.60@tcp [ 2388.014489] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 15:57:40 (1788638260) [ 2388.147603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2388.210206] LNet: 115366:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2388.227042] LNetError: 115366:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2388.258221] LNet: Removed LNI 192.168.204.60@tcp [ 2388.302318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2388.355653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 2388.388708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 2395.856700] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 15:57:48 (1788638268) [ 2395.960992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2395.992135] LNet: 115948:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2396.017292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2396.093729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2396.512393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2396.535070] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2396.603590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 2397.274899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2397.320865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2397.454694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2397.488630] LNet: 115986:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2397.531089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2397.623286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2398.128736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2398.169911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2398.235905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 2398.767672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2398.816759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2406.305433] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 15:57:59 (1788638279) [ 2406.431252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2406.496518] LNet: 116581:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2406.561409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2406.689402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2406.715299] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 2407.675725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2407.716924] LNet: Added LNI 192.168.204.60@tcp2 [8/256/0/180] [ 2408.695684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2408.755831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.60@tcp [ 2408.950925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2408.970795] LNetError: 116867:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2408.976815] LNet: Removed LNI 192.168.204.60@tcp [ 2408.988512] LNet: Removed LNI 192.168.204.60@tcp2 [ 2409.003946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2409.082681] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 2409.092434] LNet: Added LNI 192.168.204.60@tcp2 [8/256/0/180] [ 2409.105791] LNet: 116873:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2409.799072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.60@tcp [ 2410.087409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2410.117468] LNetError: 116910:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2410.125119] LNetError: 116910:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2410.130985] LNet: Removed LNI 192.168.204.60@tcp [ 2410.138820] LNet: Removed LNI 192.168.204.60@tcp2 [ 2410.167816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2410.268329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2410.291078] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 2410.987537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2411.007785] LNet: Added LNI 192.168.204.60@tcp2 [8/256/0/180] [ 2411.860560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.60@tcp [ 2411.940125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2412.050596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.60@tcp [ 2412.487382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2412.523763] LNet: 117194:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2412.531503] LNet: 117194:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 2412.540276] LNetError: 117194:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2412.549800] LNetError: 117194:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2412.559723] LNet: Removed LNI 192.168.204.60@tcp [ 2413.618621] LNet: Removed LNI 192.168.204.60@tcp2 [ 2413.642115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2413.728200] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 2413.736916] LNet: Added LNI 192.168.204.60@tcp2 [8/256/0/180] [ 2413.743390] LNet: 117200:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2414.179335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.60@tcp [ 2414.451688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2414.471701] LNet: Removed LNI 192.168.204.60@tcp [ 2415.529883] LNet: Removed LNI 192.168.204.60@tcp2 [ 2415.593061] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2415.671925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2415.691580] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 2416.710884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2416.729423] LNet: Added LNI 192.168.204.60@tcp2 [8/256/0/180] [ 2417.825864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2417.910306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.60@tcp [ 2418.197702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2418.220566] LNetError: 117517:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2418.225051] LNetError: 117517:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 2419.231415] LNet: Removed LNI 192.168.204.60@tcp [ 2419.244930] LNet: Removed LNI 192.168.204.60@tcp2 [ 2419.291318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2419.376872] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 2419.384725] LNet: Added LNI 192.168.204.60@tcp2 [8/256/0/180] [ 2419.391206] LNet: 117524:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2420.321853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.60@tcp [ 2420.649906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2420.704466] LNet: 117561:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2420.711495] LNet: 117561:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 2421.727201] LNet: Removed LNI 192.168.204.60@tcp [ 2421.744799] LNet: Removed LNI 192.168.204.60@tcp2 [ 2421.766598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2421.814956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2421.838634] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 2423.052899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2423.083771] LNet: Added LNI 192.168.204.60@tcp2 [8/256/0/180] [ 2424.616436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.60@tcp [ 2424.753123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2424.924508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.60@tcp [ 2425.312028] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2426.399194] LNet: Removed LNI 192.168.204.60@tcp [ 2426.403938] LNetError: 117846:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2426.421245] LNetError: 117846:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 2426.452154] LNet: Removed LNI 192.168.204.60@tcp2 [ 2426.539565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2426.690035] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 2426.712186] LNet: Added LNI 192.168.204.60@tcp2 [8/256/0/180] [ 2426.722755] LNet: 117852:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2427.662042] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.60@tcp [ 2427.883405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2427.932485] LNet: Removed LNI 192.168.204.60@tcp [ 2428.969419] LNet: Removed LNI 192.168.204.60@tcp2 [ 2429.008647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2429.081964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2429.125087] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 2429.898499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2429.929730] LNet: Added LNI 192.168.204.60@tcp2 [8/256/0/180] [ 2431.273408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2431.313969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2431.367368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.60@tcp [ 2431.709767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2432.800717] LNet: Removed LNI 192.168.204.60@tcp [ 2432.827912] LNet: Removed LNI 192.168.204.60@tcp2 [ 2432.939023] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2433.070210] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 2433.076810] LNet: Added LNI 192.168.204.60@tcp2 [8/256/0/180] [ 2433.084271] LNet: 118192:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2434.056866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.60@tcp [ 2434.525157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2435.615236] LNet: Removed LNI 192.168.204.60@tcp [ 2435.631525] LNet: Removed LNI 192.168.204.60@tcp2 [ 2435.659715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2435.726334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2435.757123] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 2436.581405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2436.607104] LNet: Added LNI 192.168.204.60@tcp2 [8/256/0/180] [ 2437.694914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.60@tcp [ 2437.772972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2437.832915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2437.899492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.60@tcp [ 2438.477758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2438.514885] LNet: 118546:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2438.520525] LNet: 118546:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 2439.583169] LNet: Removed LNI 192.168.204.60@tcp [ 2440.677734] LNet: Removed LNI 192.168.204.60@tcp2 [ 2440.750522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2440.893292] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 2440.925360] LNet: Added LNI 192.168.204.60@tcp2 [8/256/0/180] [ 2441.749522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.60@tcp [ 2450.689422] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 2453.289738] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 15:58:45 (1788638325) [ 2453.466635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2453.498610] LNetError: 119306:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2453.508526] LNetError: 119306:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [ 2453.518068] LNet: Removed LNI 192.168.204.60@tcp [ 2454.578217] LNet: Removed LNI 192.168.204.60@tcp2 [ 2454.610492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2455.043847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2455.115687] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 2456.109883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2456.131966] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2457.084828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2457.117691] LNet: Removed LNI 10.1.2.3@tcp [ 2457.130923] LNet: Removed LNI 192.168.204.60@tcp [ 2457.147854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2457.273259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 2457.291627] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 2457.300580] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2457.719282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 2457.746791] LNet: Removed LNI 192.168.204.60@tcp [ 2457.763950] LNet: Removed LNI 10.1.2.3@tcp [ 2464.973816] Lustre: DEBUG MARKER: == sanity-lnet test 475: per-operation latency stats populate and reset ========================================================== 15:58:57 (1788638337) [ 2465.149967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2465.231192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2465.322052] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2465.353180] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 2465.932496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2465.955853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2466.006493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.60@tcp [ 2466.039857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.60@tcp [ 2466.065864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.60@tcp [ 2466.101441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.60@tcp [ 2466.138469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.60@tcp [ 2466.163693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.60@tcp [ 2466.207664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.60@tcp [ 2466.252753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.60@tcp [ 2466.301570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.60@tcp [ 2466.342405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.60@tcp [ 2466.565471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2473.867876] Lustre: DEBUG MARKER: == sanity-lnet test 476: lnetctl set latency_stats toggles capture ========================================================== 15:59:06 (1788638346) [ 2473.938762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2473.956789] LNet: 120924:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2473.962179] LNet: 120924:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 2474.989928] LNet: Removed LNI 192.168.204.60@tcp [ 2475.026357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2475.125642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2475.168252] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 2475.892043] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 0 [ 2475.916631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2475.947423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.60@tcp [ 2475.987660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.60@tcp [ 2476.041528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.60@tcp [ 2476.080425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.60@tcp [ 2476.117988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.60@tcp [ 2476.145369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.60@tcp [ 2476.186608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.60@tcp [ 2476.244453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.60@tcp [ 2476.304660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.60@tcp [ 2476.353156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.60@tcp [ 2476.463499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2476.515427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.60@tcp [ 2476.550993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.60@tcp [ 2476.608398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.60@tcp [ 2476.662197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.60@tcp [ 2476.700341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.60@tcp [ 2476.771279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.60@tcp [ 2476.831416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.60@tcp [ 2476.890419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.60@tcp [ 2476.962155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.60@tcp [ 2476.991271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.60@tcp [ 2483.162701] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 15:59:16 (1788638356) [ 2483.391023] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2484.460357] LNet: Removed LNI 192.168.204.60@tcp [ 2484.493088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2484.639882] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2484.779173] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2484.781643] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2484.785643] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2487.140627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2487.186983] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 2487.822782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2487.841755] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2489.829940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.60@tcp [ 2489.872368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 2489.944843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.204.60@tcp -m PUT -e local_timeout -r 1 [ 2490.023644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 2490.113692] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2490.121493] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2490.336440] LNetError: 121750:0:(lib-move.c:4794:lnet_parse()) 192.168.204.60@tcp, src 192.168.204.60@tcp: Dropping PUT (error -108 looking up sender) [ 2491.360552] LNetError: 122209:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2491.372304] LNetError: 122209:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 2491.384481] LNet: Removed LNI 10.1.2.3@tcp [ 2492.455341] LNet: Removed LNI 192.168.204.60@tcp [ 2493.004922] Key type .llcrypt unregistered [ 2493.007425] Key type ._llcrypt unregistered [ 2506.190446] Key type ._llcrypt registered [ 2506.198551] Key type .llcrypt registered [ 2507.716384] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 15:59:40 (1788638380) [ 2508.088462] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2508.100899] alg: No test for adler32 (adler32-zlib) [ 2509.037758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2509.520677] LNet: 123063:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2510.275680] Key type .llcrypt unregistered [ 2510.281216] Key type ._llcrypt unregistered [ 2518.285180] Key type ._llcrypt registered [ 2518.287762] Key type .llcrypt registered [ 2518.526714] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2518.552333] alg: No test for adler32 (adler32-zlib) [ 2519.473268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2519.507355] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 2527.985779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.160@tcp [ 2528.228954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 2528.994395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.204.160@tcp [ 2529.271297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2529.307902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2529.334441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2529.369036] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2529.407956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2529.440170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2529.474507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2529.524352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2529.564942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2529.589302] LNet: 123512:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2529.610498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2529.638938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2529.682634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2529.744591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2529.759501] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.160@tcp [ 2529.788726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2529.835486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2530.439302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2530.688273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2530.718868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2530.748304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2530.791419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2530.831515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2530.906826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2530.960808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2531.012609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2531.062932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2531.079230] LNet: 123631:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2531.106617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2531.163925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2531.220617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2531.264553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2531.312937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2531.360086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2532.242817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.204.160@tcp [ 2532.599317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2532.663655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2532.715400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2532.769348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2532.816725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2532.856441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2532.900729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2532.954513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2533.013144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2533.044325] LNet: 123751:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2533.089790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2533.132106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2533.170163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2533.214859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2533.281894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2533.339694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2534.083331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2534.309591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2534.344659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2534.403444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2534.439521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2534.483735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2534.524352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2534.557536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2534.618427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2534.664136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2534.712545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2534.739417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2534.770714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2534.811396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2534.854163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2534.892108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2535.190217] LNet: 123916:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2535.195567] LNetError: 123916:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2536.231292] LNet: Removed LNI 192.168.204.60@tcp [ 2536.742828] Key type .llcrypt unregistered [ 2536.745500] Key type ._llcrypt unregistered [ 2554.824881] Key type ._llcrypt registered [ 2554.828652] Key type .llcrypt registered [ 2556.536442] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 16:00:28 (1788638428) [ 2556.993250] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2557.014102] alg: No test for adler32 (adler32-zlib) [ 2557.915870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2558.381472] LNet: 124800:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2559.198900] Key type .llcrypt unregistered [ 2559.208922] Key type ._llcrypt unregistered [ 2566.446220] Key type ._llcrypt registered [ 2566.454877] Key type .llcrypt registered [ 2567.017958] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2567.050638] alg: No test for adler32 (adler32-zlib) [ 2567.957123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2567.982111] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 2575.603399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.160@tcp [ 2585.391212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2585.457509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2585.481095] LNet: Added LNI 192.168.204.60@tcp1 [8/256/0/180] [ 2586.327411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 2587.132254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.204.160@tcp [ 2587.423759] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.160@tcp [ 2587.499400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2587.533322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2587.565298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2587.605347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2587.641787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2587.688895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2587.719643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2587.768721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2587.816293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2587.863927] LNet: 125464:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2587.928234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2587.995445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2588.057641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2588.120240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2588.192407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2588.242348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2589.061755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2589.391506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2589.424992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2589.453772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2589.483647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2589.511843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2589.546921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2589.576336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2589.620542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2589.656126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2589.678705] LNet: 125584:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2589.699859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2589.733867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2589.774816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2589.812921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2589.854919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2589.899259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2590.617775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.204.160@tcp [ 2590.854408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2590.883821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2590.916755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2590.938576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2590.965314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2591.024339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2591.063487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2591.100378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2591.139867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2591.163607] LNet: 125703:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2591.191723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2591.225362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2591.260751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2591.292145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2591.317654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2591.354306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2592.062350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2592.329357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2592.373909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2592.421995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2592.464676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2592.501988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2592.530829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2592.562423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2592.601621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2592.630321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2592.677657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2592.733861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2592.770372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2592.812626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2592.851191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2592.902504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2598.416115] LNet: 125903:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2598.426720] LNetError: 125903:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2598.435200] LNet: Removed LNI 192.168.204.60@tcp [ 2598.451366] LNet: Removed LNI 192.168.204.60@tcp1 [ 2598.935704] Key type .llcrypt unregistered [ 2598.943593] Key type ._llcrypt unregistered [ 2615.790557] Key type ._llcrypt registered [ 2615.792485] Key type .llcrypt registered [ 2617.026806] Lustre: DEBUG MARKER: == sanity-lnet test 525: keep reachable peer on local net delete ========================================================== 16:01:30 (1788638490) [ 2617.385450] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2617.396693] alg: No test for adler32 (adler32-zlib) [ 2618.335640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2618.381658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2618.396821] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 2619.190887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2619.204464] LNet: Added LNI 192.168.204.60@tcp2 [8/256/0/180] [ 2620.030756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.60@tcp [ 2620.094359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp [ 2621.151263] LNet: Removed LNI 192.168.204.60@tcp [ 2628.228641] Lustre: DEBUG MARKER: == sanity-lnet test 550: DD-off: keep route up when gw reachable via another local NI ========================================================== 16:01:41 (1788638501) [ 2629.668212] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_550 Need at least 2 local interfaces for multi-rail [ 2631.435975] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 16:01:44 (1788638504) [ 2638.905792] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 16:01:51 (1788638511) [ 2646.317728] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 16:01:59 (1788638519) [ 2654.667979] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 16:02:07 (1788638527) [ 2661.987154] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 16:02:15 (1788638535) [ 2668.970109] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 16:02:22 (1788638542) [ 2676.492788] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 16:02:29 (1788638549) [ 2683.530575] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 16:02:36 (1788638556) [ 2692.105687] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 16:02:44 (1788638564) [ 2699.401088] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 16:02:52 (1788638572) [ 2707.050819] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 16:02:59 (1788638579) [ 2715.155529] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 16:03:08 (1788638588) [ 2723.340818] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 16:03:16 (1788638596) [ 2730.326957] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 16:03:23 (1788638603) [ 2738.252594] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 16:03:30 (1788638610) [ 2745.550893] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 16:03:38 (1788638618) [ 2754.028874] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 16:03:47 (1788638627) [ 2762.768480] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 16:03:55 (1788638635) [ 2763.012902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2763.053993] LNet: 137594:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2763.068278] LNetError: 137594:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2763.087060] LNet: Removed LNI 192.168.204.60@tcp2 [ 2763.107927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2763.271570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2763.297470] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 2764.673397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2764.704682] LNet: 137736:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2764.723229] LNetError: 137736:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2764.738470] LNet: Removed LNI 192.168.204.60@tcp [ 2764.775227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2764.848637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2764.880235] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 2772.018782] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 16:04:05 (1788638645) [ 2778.480922] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 16:04:11 (1788638651) [ 2786.480790] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 16:04:18 (1788638658) [ 2793.862233] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 16:04:26 (1788638666) [ 2801.663976] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 16:04:34 (1788638674) [ 2808.852891] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 16:04:41 (1788638681) [ 2816.716656] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 16:04:49 (1788638689) [ 2824.427332] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 16:04:57 (1788638697) [ 2832.990584] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 16:05:05 (1788638705) [ 2842.412205] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 16:05:14 (1788638714) [ 2850.075641] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 16:05:22 (1788638722) [ 2859.369875] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 16:05:32 (1788638732) [ 2867.252554] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 2627 sec ========== 16:05:40 (1788638740) [ 2867.740774] LNet: 145201:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2867.756850] LNetError: 145201:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2867.811701] LNet: Removed LNI 192.168.204.60@tcp [ 2868.894749] Key type .llcrypt unregistered [ 2868.902372] Key type ._llcrypt unregistered [ 2882.003326] Key type ._llcrypt registered [ 2882.005120] Key type .llcrypt registered [ 2882.509245] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2882.523843] alg: No test for adler32 (adler32-zlib) [ 2883.634362] Lustre: Lustre: Build Version: 2.17.58_2_g6837bc1 [ 2883.917057] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 2885.594095] Key type lgssc registered [ 2886.747335] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2952.700844] Lustre: Mounted lustre-client - version 2.17.58_2_g6837bc1 [ 2957.869993] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2966.041274] Lustre: Unmounted lustre-client [ 3004.317383] Key type lgssc unregistered [ 3004.601829] LNet: 148152:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3004.616177] LNetError: 148152:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3004.662782] LNet: Removed LNI 192.168.204.60@tcp [ 3005.350180] Key type .llcrypt unregistered [ 3005.356129] Key type ._llcrypt unregistered