[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 519089137 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001013] APIC: Switch to symmetric I/O mode setup [ 0.003164] x2apic enabled [ 0.004007] Switched APIC routing to physical x2apic. [ 0.005012] kvm-guest: setup PV IPIs [ 0.008000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008034] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009019] pid_max: default: 32768 minimum: 301 [ 0.010184] LSM: Security Framework initializing [ 0.011090] Yama: becoming mindful. [ 0.012051] SELinux: Initializing. [ 0.014067] *** VALIDATE selinux *** [ 0.022714] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027286] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028145] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029126] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030120] *** VALIDATE tmpfs *** [ 0.031464] *** VALIDATE proc *** [ 0.033015] *** VALIDATE cgroup *** [ 0.034009] *** VALIDATE cgroup2 *** [ 0.035280] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.037061] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.038010] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.039028] Spectre V2 : User space: Vulnerable [ 0.040009] Speculative Store Bypass: Vulnerable [ 0.043143] debug: unmapping init [mem 0xffffffffa1459000-0xffffffffa1460fff] [ 0.045146] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.046746] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.047028] ... version: 2 [ 0.048011] ... bit width: 48 [ 0.048940] ... generic registers: 4 [ 0.049018] ... value mask: 0000ffffffffffff [ 0.050015] ... max period: 00007fffffffffff [ 0.051017] ... fixed-purpose events: 3 [ 0.052013] ... event mask: 000000070000000f [ 0.053291] rcu: Hierarchical SRCU implementation. [ 0.055692] smp: Bringing up secondary CPUs ... [ 0.056679] x86: Booting SMP configuration: [ 0.057034] .... node #0, CPUs: #1 #2 #3 [ 0.061459] smp: Brought up 1 node, 4 CPUs [ 0.063016] smpboot: Max logical packages: 1 [ 0.064020] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.138143] node 0 deferred pages initialised in 70ms [ 0.142221] devtmpfs: initialized [ 0.143403] x86/mm: Memory block size: 128MB [ 0.146661] gcov: version magic: 0x41383552 [ 0.148358] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.149088] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.150282] pinctrl core: initialized pinctrl subsystem [ 0.151538] [ 0.152009] ************************************************************* [ 0.153014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.154066] ** ** [ 0.155013] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.156013] ** ** [ 0.157012] ** This means that this kernel is built to expose internal ** [ 0.158013] ** IOMMU data structures, which may compromise security on ** [ 0.159009] ** your system. ** [ 0.160014] ** ** [ 0.161011] ** If you see this message and you are not debugging the ** [ 0.162007] ** kernel, report this immediately to your vendor! ** [ 0.163013] ** ** [ 0.164013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.165018] ************************************************************* [ 0.166951] NET: Registered protocol family 16 [ 0.167687] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.168066] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.169076] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.170864] cpuidle: using governor menu [ 0.172017] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.173649] PCI: Using configuration type 1 for base access [ 0.174145] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.183120] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.185067] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.190251] cryptd: max_cpu_qlen set to 1000 [ 0.192206] ACPI: Added _OSI(Module Device) [ 0.193000] ACPI: Added _OSI(Processor Device) [ 0.195020] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.197012] ACPI: Added _OSI(Processor Aggregator Device) [ 0.203518] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.210833] ACPI: Interpreter enabled [ 0.211077] ACPI: PM: (supports S0 S3 S4 S5) [ 0.212011] ACPI: Using IOAPIC for interrupt routing [ 0.213079] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.214303] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.227374] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.228041] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.229019] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.230115] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.232508] acpiphp: Slot [2] registered [ 0.233146] acpiphp: Slot [5] registered [ 0.234160] acpiphp: Slot [6] registered [ 0.235140] acpiphp: Slot [3] registered [ 0.236085] acpiphp: Slot [4] registered [ 0.237102] acpiphp: Slot [7] registered [ 0.238129] acpiphp: Slot [8] registered [ 0.239108] acpiphp: Slot [9] registered [ 0.240080] acpiphp: Slot [10] registered [ 0.241144] acpiphp: Slot [11] registered [ 0.242106] acpiphp: Slot [12] registered [ 0.243099] acpiphp: Slot [13] registered [ 0.244127] acpiphp: Slot [14] registered [ 0.245113] acpiphp: Slot [15] registered [ 0.246262] acpiphp: Slot [16] registered [ 0.247167] acpiphp: Slot [17] registered [ 0.248139] acpiphp: Slot [18] registered [ 0.249157] acpiphp: Slot [19] registered [ 0.250141] acpiphp: Slot [20] registered [ 0.251134] acpiphp: Slot [21] registered [ 0.252182] acpiphp: Slot [22] registered [ 0.253000] acpiphp: Slot [23] registered [ 0.253000] acpiphp: Slot [24] registered [ 0.253107] acpiphp: Slot [25] registered [ 0.254000] acpiphp: Slot [26] registered [ 0.256117] acpiphp: Slot [27] registered [ 0.258164] acpiphp: Slot [28] registered [ 0.259213] acpiphp: Slot [29] registered [ 0.261158] acpiphp: Slot [30] registered [ 0.262162] acpiphp: Slot [31] registered [ 0.264172] PCI host bridge to bus 0000:00 [ 0.266019] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.268023] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.270027] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.273023] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.275025] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.278030] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.279311] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.282840] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.286596] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.294015] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.299044] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.301017] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.304016] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.306015] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.308474] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.311914] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.314038] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.316922] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.321099] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.332012] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.333000] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.338227] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.347017] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.361026] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.383019] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.397000] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.411017] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.420014] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.434018] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.449734] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.452385] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.454429] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.456436] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.460193] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.464037] iommu: Default domain type: Passthrough [ 0.466547] SCSI subsystem initialized [ 0.468198] ACPI: bus type USB registered [ 0.470118] usbcore: registered new interface driver usbfs [ 0.471068] usbcore: registered new interface driver hub [ 0.473078] usbcore: registered new device driver usb [ 0.475240] pps_core: LinuxPPS API ver. 1 registered [ 0.478010] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.481060] PTP clock support registered [ 0.484099] EDAC MC: Ver: 3.0.0 [ 0.485490] PCI: Using ACPI for IRQ routing [ 0.486767] NetLabel: Initializing [ 0.487009] NetLabel: domain hash size = 128 [ 0.488008] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.489087] NetLabel: unlabeled traffic allowed by default [ 0.490365] vgaarb: loaded [ 0.492590] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.494010] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.498348] clocksource: Switched to clocksource kvm-clock [ 0.580223] VFS: Disk quotas dquot_6.6.0 [ 0.581166] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.582403] *** VALIDATE ramfs *** [ 0.583059] *** VALIDATE hugetlbfs *** [ 0.584483] pnp: PnP ACPI init [ 0.586426] pnp: PnP ACPI: found 6 devices [ 0.602205] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.604112] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.605334] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.606866] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.608433] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.609985] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.612209] NET: Registered protocol family 2 [ 0.613842] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.617194] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.620536] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.626107] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.628816] TCP: Hash tables configured (established 65536 bind 65536) [ 0.630953] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.633207] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.635120] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.637527] NET: Registered protocol family 1 [ 0.639494] RPC: Registered named UNIX socket transport module. [ 0.640606] RPC: Registered udp transport module. [ 0.641432] RPC: Registered tcp transport module. [ 0.642222] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.643430] NET: Registered protocol family 44 [ 0.644302] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.645320] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.646740] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.648711] PCI: CLS 0 bytes, default 64 [ 0.650205] Unpacking initramfs... [ 2.237926] debug: unmapping init [mem 0xffff8fba3cc64000-0xffff8fba3ffcffff] [ 2.242180] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.244251] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.246803] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.587686] Initialise system trusted keyrings [ 3.589940] Key type blacklist registered [ 3.593095] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.607899] zbud: loaded [ 3.613859] *** VALIDATE nfs *** [ 3.615896] *** VALIDATE nfs4 *** [ 3.617729] pstore: using deflate compression [ 3.623452] Platform Keyring initialized [ 3.909912] NET: Registered protocol family 38 [ 3.913296] Key type asymmetric registered [ 3.916328] Asymmetric key parser 'x509' registered [ 3.918460] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.925864] io scheduler mq-deadline registered [ 3.931016] io scheduler kyber registered [ 3.937519] io scheduler bfq registered [ 3.940966] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.946424] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.949866] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.953829] ACPI: Power Button [PWRF] [ 3.963402] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.977862] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 4.004850] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 4.034697] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 4.082855] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 4.108765] Non-volatile memory driver v1.3 [ 4.113178] Linux agpgart interface v0.103 [ 4.180396] virtio_blk virtio1: [vda] 150408 512-byte logical blocks (77.0 MB/73.4 MiB) [ 4.184513] vda: detected capacity change from 0 to 77008896 [ 4.238119] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 4.244661] vdb: detected capacity change from 0 to 1073741824 [ 4.273812] libphy: Fixed MDIO Bus: probed [ 4.325608] usbcore: registered new interface driver usbserial_generic [ 4.330266] usbserial: USB Serial support registered for generic [ 4.334221] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 4.339885] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 4.345924] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 4.353722] mousedev: PS/2 mouse device common for all mice [ 4.360582] rtc_cmos 00:05: RTC can wake from S4 [ 4.366563] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 4.367681] rtc_cmos 00:05: registered as rtc0 [ 4.378246] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 4.381590] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 4.387807] intel_pstate: CPU model not supported [ 4.393632] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 4.399285] hid: raw HID events driver (C) Jiri Kosina [ 4.401941] usbcore: registered new interface driver usbhid [ 4.404302] usbhid: USB HID core driver [ 4.406217] drop_monitor: Initializing network drop monitor service [ 4.409100] Initializing XFRM netlink socket [ 4.411381] NET: Registered protocol family 10 [ 4.416967] Segment Routing with IPv6 [ 4.418706] NET: Registered protocol family 17 [ 4.420590] mpls_gso: MPLS GSO support [ 4.438413] RAS: Correctable Errors collector initialized. [ 4.445067] AVX version of gcm_enc/dec engaged. [ 4.448677] AES CTR mode by8 optimization enabled [ 4.596784] sched_clock: Marking stable (4596765326, 0)->(5594303587, -997538261) [ 4.600321] registered taskstats version 1 [ 4.602481] Loading compiled-in X.509 certificates [ 4.605633] zswap: loaded using pool lzo/zbud [ 4.708708] Key type big_key registered [ 4.728480] Key type encrypted registered [ 4.730362] ima: No TPM chip found, activating TPM-bypass! [ 4.732501] ima: Allocated hash algorithm: sha1 [ 4.734522] ima: No architecture policies found [ 4.736355] evm: Initialising EVM extended attributes: [ 4.738395] evm: security.selinux [ 4.739780] evm: security.ima [ 4.740980] evm: security.capability [ 4.743099] evm: HMAC attrs: 0x1 [ 4.745841] rtc_cmos 00:05: setting system clock to 2026-09-22 17:01:13 UTC (1790096473) [ 4.754233] debug: unmapping init [mem 0xffffffffa2403000-0xffffffffa25fffff] [ 4.757599] debug: unmapping init [mem 0xffffffffa1182000-0xffffffffa1458fff] [ 4.769355] Write protecting the kernel read-only data: 28672k [ 4.772673] debug: unmapping init [mem 0xffffffff9f803000-0xffffffff9f9fffff] [ 4.776219] debug: unmapping init [mem 0xffffffffa0114000-0xffffffffa01fffff] [ 4.818654] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.831732] systemd[1]: Detected virtualization kvm. [ 4.834497] systemd[1]: Detected architecture x86-64. [ 4.837457] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.878643] systemd[1]: No hostname configured. [ 4.882408] systemd[1]: Set hostname to . [ 4.886262] random: systemd: uninitialized urandom read (16 bytes read) [ 4.890956] systemd[1]: Initializing machine ID from random generator. [ 5.122148] random: systemd: uninitialized urandom read (16 bytes read) [ 5.131261] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 5.142399] random: systemd: uninitialized urandom read (16 bytes read) [ 5.146267] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 5.158231] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local File Systems. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Timers. [ OK ] Listening on Journal Socket. Starting Apply Kernel Variables... [ OK ] Started Memstrack Anylazing Service. Starting Journal Service... [ OK ] Reached target Sockets. Starting Create Volatile Files and Directories... Starting Create list of required st…ce nodes for the current kernel... Starting Setup Virtual Console... [ OK ] Reached target Initrd Root Device. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 5.858386] device-mapper: uevent: version 1.0.3 [ 5.859886] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK [ 6.769457] random: fast init done ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 6.954378] virtio_net virtio0 ens2: renamed from eth0 [ 7.373852] scsi host0: ata_piix [ 7.425295] scsi host1: ata_piix [ 7.666908] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 7.674905] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 12.259749] random: crng init done [ 12.260840] random: 7 urandom warning(s) missed due to ratelimiting [ 15.495557] dracut-initqueue[590]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 17.843599] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Timers. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Swap. [ OK ] Stopped target Sockets. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 20.704987] printk: systemd: 26 output lines suppressed due to ratelimiting [ 21.666494] SELinux: Disabled at runtime. [ 21.970249] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 22.039398] systemd[1]: Detected virtualization kvm. [ 22.041909] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 25.238878] systemd[1]: initrd-switch-root.service: Succeeded. [ 25.244787] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 25.259692] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 25.264361] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 25.270245] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 25.281220] systemd[1]: Starting Journal Service... Starting Journal Service... [ 25.299573] systemd[1]: Listening on RPCbind Server Activation Socket. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice system-getty.slice. Mounting Huge Pages File System... [ OK ] Started Forward Password Requests to Wall Directory Watch. Mounting POSIX Message Queue File System... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Activating swap /dev/disk/by-label/SWAP... Mounting Kernel Debug File System... [ 25.547718] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on Process Core Dump Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. Starting Remount Root and Kernel File Systems... [ OK ] Listening on udev Control Socket. Starting Create list of required st…ce nodes for the current kernel... Starting Apply Kernel Variables... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Reached target RPC Port Mapper. [ OK ] Created slice system-sshd\x2dkeygen.slice. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. [ OK ] Started Journal Service. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Kernel Debug File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started udev Coldplug all Devices. [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ 27.074576] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 28.033953] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 28.054713] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 28.590611] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 28.819174] EDAC sbridge: Ver: 1.1.2 [ 32.254576] Key type dns_resolver registered [* ] A start job is running for Configur…-only root support (7s / no limit)[ 32.946158] NFS: Registering the id_resolver key type [ 32.948232] Key type id_resolver registered [ 32.950147] Key type id_legacy registered [** ] A start job is running for Configur…-only root support (8s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. Starting Network Manager... Starting Login Service... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... [ OK ] Started Login Service. Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg235-client login: [ 102.911190] libcfs: loading out-of-tree module taints kernel. [ 103.195246] Key type ._llcrypt registered [ 103.238329] Key type .llcrypt registered [ 103.878360] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 103.900454] alg: No test for adler32 (adler32-zlib) [ 105.533922] Lustre: Lustre: Build Version: 2.17.58_63_gba3a43f [ 106.561468] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 108.311544] Key type lgssc registered [ 110.449370] Lustre: Echo OBD driver; http://www.lustre.org/ [ 234.478505] Lustre: Mounted lustre-client - version 2.17.58_63_gba3a43f [ 239.402429] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 253.017708] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing check_logdir /tmp/testlogs/ [ 257.802271] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing yml_node [ 260.063227] Lustre: lustre-OST0000-osc-ffff8fba82b98800: disconnect after 23s idle [ 261.807582] Lustre: DEBUG MARKER: Client: 2.17.58.63 [ 263.994970] Lustre: DEBUG MARKER: MDS: 2.17.58.63 [ 266.452590] Lustre: DEBUG MARKER: OSS: 2.17.58.63 [ 268.160402] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Tue Sep 22 13:05:35 EDT 2026 [ 284.913130] Lustre: Unmounted lustre-client [ 320.763945] Key type lgssc unregistered [ 321.007254] LNet: 8428:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 321.016038] LNetError: 8428:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 321.031280] LNet: Removed LNI 192.168.202.35@tcp [ 321.664160] Key type .llcrypt unregistered [ 321.667861] Key type ._llcrypt unregistered [ 349.046324] Key type ._llcrypt registered [ 349.048548] Key type .llcrypt registered [ 349.337690] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 349.383438] alg: No test for adler32 (adler32-zlib) [ 350.403593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 350.489524] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 350.515161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 354.697737] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 354.841630] LNet: 9317:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 354.856461] LNetError: 9317:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 354.898554] LNet: Removed LNI 192.168.202.35@tcp [ 355.415343] Key type .llcrypt unregistered [ 355.428399] Key type ._llcrypt unregistered [ 363.376599] Key type ._llcrypt registered [ 363.377544] Key type .llcrypt registered [ 364.777384] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 13:07:11 (1790096831) [ 364.950773] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 364.959531] alg: No test for adler32 (adler32-zlib) [ 365.885555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 365.995901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 366.016318] LNet: 9731:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 366.056451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 366.161791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 366.194113] LNet: 9738:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 373.103672] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 13:07:20 (1790096840) [ 373.183949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 373.212675] LNet: 10310:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 373.249724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 373.354730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 380.185646] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 13:07:27 (1790096847) [ 380.487848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 380.545683] LNet: 10905:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 380.576452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 380.690362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 388.350919] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 13:07:35 (1790096855) [ 388.463202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 388.509659] LNet: 11481:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 388.533848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 388.658408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 396.772441] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 13:07:43 (1790096863) [ 396.910116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 396.941388] LNet: 12069:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 396.976359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 397.106821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 397.597841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 405.100946] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 13:07:52 (1790096872) [ 405.221967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 405.252521] LNet: 12656:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 405.274574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 405.442682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 406.093449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 406.752628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 414.241837] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 13:08:01 (1790096881) [ 414.396406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 414.475878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 414.586250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 422.131890] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 13:08:09 (1790096889) [ 422.337499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 422.371033] LNet: 13844:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 422.379634] LNet: 13844:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 422.428841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 422.511954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 422.560764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 422.993981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 423.021923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 423.380094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 423.431252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 424.066523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 424.116909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 424.652964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 424.688892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 425.427766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 425.468433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 425.950393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 425.991500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 426.483261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 426.522859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 426.996618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 427.070862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 427.637416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 427.695523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 428.360823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 428.412364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 428.963622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 429.064171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 429.747901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 429.804467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 436.970376] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 13:08:24 (1790096904) [ 437.099878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 437.172861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 437.307526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 437.347828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 443.723572] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 13:08:31 (1790096911) [ 443.854780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 443.931222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 444.022563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 444.074750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 451.689184] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 13:08:39 (1790096919) [ 451.812529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 451.849580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 451.957042] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 452.025295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 458.753413] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 13:08:46 (1790096926) [ 458.920527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 458.966775] LNet: 16281:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 458.975681] LNet: 16281:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 459.031737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 459.143583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 459.181313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 464.222402] hrtimer: interrupt took 2598004 ns [ 468.328304] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 13:08:55 (1790096935) [ 468.590737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 468.731739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 468.816843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 468.866557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 476.904882] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 13:09:03 (1790096943) [ 477.269139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 477.327456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 477.425701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 477.494345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 484.896286] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 13:09:12 (1790096952) [ 485.009871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 485.102250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 485.262562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 485.686657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 492.165862] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 13:09:19 (1790096959) [ 492.255863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 492.343561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 492.541420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 492.744282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 500.951508] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 13:09:27 (1790096967) [ 501.113760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 501.194185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 501.425246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 501.705297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 509.853903] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 13:09:36 (1790096976) [ 509.977351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 510.049561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 510.689752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 511.060612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 519.336101] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 13:09:46 (1790096986) [ 519.440402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 519.576736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 519.689691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 519.740401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 527.733142] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 13:09:54 (1790096994) [ 527.881250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 527.937130] LNet: 21116:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 527.951258] LNet: 21116:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 7 previous similar messages [ 527.993719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 528.156809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 528.212448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 535.315714] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 13:10:02 (1790097002) [ 535.483054] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 535.548438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 535.632415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 543.563633] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 13:10:10 (1790097010) [ 543.671475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 543.713914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 543.841341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 550.329378] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 13:10:17 (1790097017) [ 550.521689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 550.620221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 550.691677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 550.998508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 557.565877] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 13:10:24 (1790097024) [ 557.794255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 557.870748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 557.998330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 558.062361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 565.421535] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 13:10:32 (1790097032) [ 565.529344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 565.615777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 565.752984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 565.807863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 573.272730] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 13:10:40 (1790097040) [ 573.410608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 573.519530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 573.661327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 573.744705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 581.678536] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 13:10:48 (1790097048) [ 581.851706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 581.937957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 582.096822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 582.176027] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 590.955614] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 13:10:57 (1790097057) [ 591.038574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 591.115015] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 591.249233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 591.384212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 591.430826] LNetError: 25848:0:(peer.c:2366:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 591.547434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 591.683673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 600.556544] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 13:11:06 (1790097066) [ 600.709221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 600.784424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 600.936404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.z2hqZpa4g6 [ 601.016870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.z2hqZpa4g6 [ 601.065553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.z2hqZpa4g6 [ 610.289654] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 13:11:17 (1790097077) [ 610.519074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 610.648971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 610.726797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 610.757775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 610.804113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 617.825557] Lustre: DEBUG MARKER: == sanity-lnet test 40: LNetAddPeer merges a pre-existing non-primary peer NID ========================================================== 13:11:24 (1790097084) [ 617.973458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 618.044479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 618.215624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 618.264164] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 620.329847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.35@tcp [ 622.239653] Lustre: Lustre: Build Version: 2.17.58_63_gba3a43f [ 624.423329] Key type lgssc registered [ 626.551065] Lustre: Echo OBD driver; http://www.lustre.org/ [ 631.640417] LNetError: 27607:0:(lib-move.c:2151:lnet_handle_find_routed_path()) peer 192.168.202.35@tcp99 has no available nets [ 635.194372] Key type lgssc unregistered [ 635.734257] LNetError: 28726:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 635.762436] LNet: Removed LNI 192.168.202.35@tcp [ 636.698639] Key type .llcrypt unregistered [ 636.711550] Key type ._llcrypt unregistered [ 645.617561] Key type ._llcrypt registered [ 645.634174] Key type .llcrypt registered [ 647.626717] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 13:11:54 (1790097114) [ 648.277536] Key type .llcrypt unregistered [ 648.280340] Key type ._llcrypt unregistered [ 655.937356] Key type ._llcrypt registered [ 655.939799] Key type .llcrypt registered [ 656.134978] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 656.157625] alg: No test for adler32 (adler32-zlib) [ 657.097311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 657.149975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 657.169792] LNet: 29648:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=16 less than recommended minimum 512 [ 657.176351] LNet: 29648:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 [ 657.183951] LNet: 29648:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 [ 657.196974] LNet: Message forwarding will be enabled in 2s [ 660.255181] LNet: Message forwarding enabled [ 660.792141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 660.831218] LNet: Message forwarding will stop in 2s to 6s [ 664.353666] LNet: Message forwarding disabled [ 665.430683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 665.461897] LNet: 29746:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=16 less than recommended minimum 512 [ 665.472422] LNet: 29746:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 [ 665.482689] LNet: 29746:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 [ 665.497887] LNet: Message forwarding will be enabled in 2s [ 665.707913] LNet: 29759:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 666.407484] Key type .llcrypt unregistered [ 666.409896] Key type ._llcrypt unregistered [ 679.698609] Key type ._llcrypt registered [ 679.705025] Key type .llcrypt registered [ 681.429693] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 13:12:28 (1790097148) [ 682.001847] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 682.023686] alg: No test for adler32 (adler32-zlib) [ 683.020463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 683.124775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 690.163969] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 13:12:37 (1790097157) [ 690.584720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 690.611359] LNet: 31163:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 690.648258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 690.730753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 690.772554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 690.800451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 690.850543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 690.909175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 690.962985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 691.002040] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 691.046975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 691.090411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 691.160976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 691.215385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 691.278734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 691.333887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 691.390949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 691.437713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 691.475741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 691.516742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 691.556672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 691.616401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 691.646180] LNetError: 31206:0:(router.c:625:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 699.070170] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 13:12:46 (1790097166) [ 699.228257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 699.261706] LNet: 31774:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 699.315528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 699.386638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 706.988830] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 13:12:53 (1790097173) [ 709.303780] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg235-server [ 711.505684] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 13:12:58 (1790097178) [ 711.723552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 711.756694] LNet: 32712:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 711.787539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 711.838440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 711.860513] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 712.620660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.202.[36-40]@tcp [ 712.640638] LNet: 32845:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 712.721064] LNetError: 32716:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.202.36@tcp is being used as a gateway but routing feature is not turned on [ 712.751246] LNetError: 32716:0:(lib-lnet.h:1281:lnet_set_route_aliveness()) route to tcp1 through 192.168.202.36@tcp has gone from up to down [ 720.369229] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 13:13:07 (1790097187) [ 722.542341] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg235-server [ 724.448617] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 13:13:11 (1790097191) [ 724.567228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 724.618924] LNet: 33775:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 724.632851] LNetError: 33775:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 724.694330] LNet: Removed LNI 192.168.202.35@tcp [ 724.745089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 724.907948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 725.007025] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 726.266444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.202.[36-41/2]@tcp [ 726.302151] LNet: 33898:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 726.317229] LNet: 33898:0:(router.c:729:lnet_add_route()) Skipped 4 previous similar messages [ 726.340477] LNetError: 33785:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.202.36@tcp is being used as a gateway but routing feature is not turned on [ 726.354486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.202.[36-41/2]@tcp [ 726.367058] LNetError: 33785:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 726.367085] LNetError: 33785:0:(lib-lnet.h:1281:lnet_set_route_aliveness()) route to tcp103 through 192.168.202.36@tcp has gone from up to down [ 726.367089] LNetError: 33785:0:(lib-lnet.h:1281:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 734.414560] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 13:13:21 (1790097201) [ 734.558092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 734.585942] LNet: 34484:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 734.599536] LNetError: 34484:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 734.621140] LNet: Removed LNI 192.168.202.35@tcp [ 734.692949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 734.800528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 734.870477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 734.909530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 735.051066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 735.118489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 735.156304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 735.244174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 735.353206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 735.422208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 735.558969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 735.667473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 735.718567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 735.798857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 735.899167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 735.982221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 736.115327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 736.251878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 736.306845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 736.358450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 736.410964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 736.464135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 736.544360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 736.582918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 736.602192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 736.643941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 736.704537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 736.734465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 736.780985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 736.847096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 736.872354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 736.904683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 736.943296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 736.990699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 737.102168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 744.579082] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 13:13:31 (1790097211) [ 747.697374] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg235-server [ 750.521385] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 13:13:36 (1790097216) [ 753.890442] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg235-server [ 757.294728] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 13:13:43 (1790097223) [ 757.466731] LNet: 35886:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 757.473879] LNet: 35886:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 11 previous similar messages [ 758.889257] Key type .llcrypt unregistered [ 758.896376] Key type ._llcrypt unregistered [ 768.380148] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 768.753519] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 768.763388] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 768.771958] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 771.421321] Key type ._llcrypt registered [ 771.423560] Key type .llcrypt registered [ 771.756844] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 771.792623] alg: No test for adler32 (adler32-zlib) [ 772.792378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 772.961324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 773.071289] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 774.470466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 774.524932] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 777.814824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 777.858634] LNet: Removed LNI 10.1.2.3@tcp [ 778.124079] LNet: 36614:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 778.131640] LNetError: 36614:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 778.151101] LNet: Removed LNI 192.168.202.35@tcp [ 778.864479] Key type .llcrypt unregistered [ 778.868302] Key type ._llcrypt unregistered [ 790.518698] Key type ._llcrypt registered [ 790.521923] Key type .llcrypt registered [ 792.022856] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 13:14:19 (1790097259) [ 792.677482] Key type .llcrypt unregistered [ 792.680137] Key type ._llcrypt unregistered [ 800.226000] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 800.588926] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 800.595900] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 800.604152] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 803.128067] Key type ._llcrypt registered [ 803.131824] Key type .llcrypt registered [ 803.389306] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 803.411114] alg: No test for adler32 (adler32-zlib) [ 804.366385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 804.421515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 804.443699] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 805.086658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 805.108164] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 807.677609] LNet: 38107:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 807.690760] LNetError: 38107:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 807.702875] LNet: Removed LNI 10.1.2.3@tcp [ 807.717886] LNet: Removed LNI 192.168.202.35@tcp [ 808.349251] Key type .llcrypt unregistered [ 808.351310] Key type ._llcrypt unregistered [ 822.022465] Key type ._llcrypt registered [ 822.025349] Key type .llcrypt registered [ 823.643059] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 13:14:50 (1790097290) [ 824.281250] Key type .llcrypt unregistered [ 824.283293] Key type ._llcrypt unregistered [ 831.350776] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 831.611425] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 831.617800] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 831.620683] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 834.316695] Key type ._llcrypt registered [ 834.318105] Key type .llcrypt registered [ 834.599672] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 834.619393] alg: No test for adler32 (adler32-zlib) [ 835.644264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 835.765631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 835.806833] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 837.719970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 837.747553] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 839.814425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 839.846909] LNet: Removed LNI 10.1.2.3@tcp [ 839.859293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 839.892535] LNet: Removed LNI 10.1.2.31@tcp [ 839.915412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 839.943053] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 840.603814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 840.620541] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 841.558846] LNet: 40028:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 841.567292] LNetError: 40028:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg:0 not found [ 841.572430] LNet: Removed LNI 10.1.2.31@tcp [ 841.582258] LNet: Removed LNI 10.1.2.3@tcp [ 842.300812] Key type .llcrypt unregistered [ 842.303248] Key type ._llcrypt unregistered [ 854.992368] Key type ._llcrypt registered [ 854.995250] Key type .llcrypt registered [ 856.073830] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 13:15:23 (1790097323) [ 856.649455] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 856.670970] alg: No test for adler32 (adler32-zlib) [ 857.612977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 857.669339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 857.687963] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 858.284177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 858.298464] LNet: 40981:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 858.302975] LNetError: 40981:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 858.314395] LNet: Removed LNI 192.168.202.35@tcp [ 858.338599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 858.394302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.202.35@tcp [ 858.424798] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 859.256388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.202.35@tcp [ 859.297732] LNet: Removed LNI 192.168.202.35@tcp [ 859.423657] LNet: 41116:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 860.234851] Key type .llcrypt unregistered [ 860.239101] Key type ._llcrypt unregistered [ 873.074601] Key type ._llcrypt registered [ 873.078164] Key type .llcrypt registered [ 874.788091] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 13:15:41 (1790097341) [ 876.953360] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg235-server [ 878.947398] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 13:15:46 (1790097346) [ 879.695645] Key type .llcrypt unregistered [ 879.697196] Key type ._llcrypt unregistered [ 886.865939] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 887.091093] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 887.095443] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 887.100378] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 889.676600] Key type ._llcrypt registered [ 889.678560] Key type .llcrypt registered [ 889.844938] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 889.862914] alg: No test for adler32 (adler32-zlib) [ 890.863166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 890.943376] LNet: Added LNI 192.168.202.35@tcp1 [8/256/0/180] [ 891.769566] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 894.068871] LNet: 42949:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 894.074841] LNetError: 42949:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 894.078083] LNet: Removed LNI 192.168.202.35@tcp1 [ 894.089149] LNet: Removed LNI 10.1.2.3@tcp2 [ 895.062919] Key type .llcrypt unregistered [ 895.065877] Key type ._llcrypt unregistered [ 908.514950] Key type ._llcrypt registered [ 908.517704] Key type .llcrypt registered [ 909.801271] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 13:16:17 (1790097377) [ 910.254516] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 910.291336] alg: No test for adler32 (adler32-zlib) [ 911.258905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 911.371147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 911.430813] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 912.472518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 912.507908] LNet: 43920:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 912.518168] LNetError: 43920:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 912.533861] LNet: Removed LNI 192.168.202.35@tcp [ 912.554191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 912.624502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 912.644943] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 919.793311] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 13:16:27 (1790097387) [ 919.933117] LNet: 44507:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 919.940126] LNetError: 44507:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 919.972491] LNet: Removed LNI 192.168.202.35@tcp [ 920.675441] Key type .llcrypt unregistered [ 920.678569] Key type ._llcrypt unregistered [ 928.038744] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 928.253774] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 928.274578] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 928.284467] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 930.870348] Key type ._llcrypt registered [ 930.872388] Key type .llcrypt registered [ 931.084535] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 931.101492] alg: No test for adler32 (adler32-zlib) [ 932.034521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 932.101763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 932.133354] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 932.959329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 932.976595] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 935.330964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 935.351634] LNet: 45255:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 935.364454] LNetError: 45255:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 935.370916] LNet: Removed LNI 10.1.2.3@tcp [ 935.382383] LNet: Removed LNI 192.168.202.35@tcp [ 935.405814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 935.495517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 935.531272] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 935.539245] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 936.162390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 936.196425] LNet: 45282:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 936.200614] LNetError: 45282:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 936.203476] LNetError: 45282:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 936.217364] LNet: Removed LNI 10.1.2.3@tcp [ 936.228463] LNet: Removed LNI 192.168.202.35@tcp [ 936.251590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 936.308683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 936.329482] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 936.335460] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 936.792603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 936.830941] LNet: Added LNI 192.168.202.35@tcp2 [8/256/0/180] [ 937.962205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 937.980222] LNet: 45444:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 937.988219] LNetError: 45444:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 937.994345] LNetError: 45444:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 938.001622] LNet: Removed LNI 10.1.2.3@tcp [ 938.004686] LNet: Removed LNI 192.168.202.35@tcp [ 938.017736] LNet: Removed LNI 192.168.202.35@tcp2 [ 938.040588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 938.110602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 938.146360] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 938.153192] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 938.163987] LNet: Added LNI 192.168.202.35@tcp2 [8/256/0/180] [ 946.472597] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 13:16:53 (1790097413) [ 946.579280] LNet: 46034:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 946.598402] LNetError: 46034:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 946.614589] LNetError: 46034:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 946.635366] LNet: Removed LNI 10.1.2.3@tcp [ 946.642978] LNet: Removed LNI 192.168.202.35@tcp [ 946.674452] LNet: Removed LNI 192.168.202.35@tcp2 [ 947.414819] Key type .llcrypt unregistered [ 947.416642] Key type ._llcrypt unregistered [ 954.454961] Key type ._llcrypt registered [ 954.456880] Key type .llcrypt registered [ 954.654549] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 954.663252] alg: No test for adler32 (adler32-zlib) [ 955.577698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 955.664410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 955.688446] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 956.576803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 956.599756] LNet: 46436:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 956.603924] LNetError: 46436:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 956.645172] LNet: Removed LNI 192.168.202.35@tcp [ 956.676390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 956.772823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 956.820431] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 957.568963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 957.608646] LNet: 46466:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 957.614378] LNetError: 46466:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 957.635078] LNet: Removed LNI 192.168.202.35@tcp [ 957.655314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 957.778994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 957.840139] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 958.658698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 958.684769] LNet: 46496:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 958.691600] LNetError: 46496:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 958.708567] LNet: Removed LNI 192.168.202.35@tcp [ 958.728894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 958.840986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 958.877971] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 959.574581] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 959.833422] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 959.842162] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 959.847357] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 961.184547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 961.205869] LNet: 46565:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 961.223023] LNetError: 46565:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 961.238787] LNet: Removed LNI 192.168.202.35@tcp [ 961.265319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 961.353627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 961.374472] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 968.476135] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 13:17:15 (1790097435) [ 968.678917] LNet: 47161:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 968.703061] LNetError: 47161:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 968.723797] LNet: Removed LNI 192.168.202.35@tcp [ 969.447232] Key type .llcrypt unregistered [ 969.450197] Key type ._llcrypt unregistered [ 976.503339] Key type ._llcrypt registered [ 976.508266] Key type .llcrypt registered [ 976.704684] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 976.721583] alg: No test for adler32 (adler32-zlib) [ 977.647759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 977.710943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 977.738901] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 978.702877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 978.720990] LNet: 47560:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 978.726453] LNetError: 47560:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 978.740587] LNet: Removed LNI 192.168.202.35@tcp [ 978.761406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 978.827677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 978.851055] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 986.240508] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 13:17:33 (1790097453) [ 986.327872] LNet: 48144:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 986.342846] LNetError: 48144:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 986.357250] LNet: Removed LNI 192.168.202.35@tcp [ 987.093528] Key type .llcrypt unregistered [ 987.097110] Key type ._llcrypt unregistered [ 993.554802] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 993.870927] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 993.888313] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 993.890268] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 996.709161] Key type ._llcrypt registered [ 996.711369] Key type .llcrypt registered [ 997.013145] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 997.038861] alg: No test for adler32 (adler32-zlib) [ 998.076941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 998.125710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 998.149229] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 999.358227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 999.375161] LNet: 48602:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 999.387061] LNetError: 48602:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 999.400970] LNet: Removed LNI 192.168.202.35@tcp [ 999.418131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 999.505692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 999.564080] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 1007.491774] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 13:17:54 (1790097474) [ 1008.008679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1008.066256] LNet: 49201:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1008.078107] LNetError: 49201:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1008.100159] LNet: Removed LNI 192.168.202.35@tcp [ 1008.176971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1008.365836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 1016.081259] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 13:18:03 (1790097483) [ 1016.297576] LNet: 49778:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1017.343782] Key type .llcrypt unregistered [ 1017.349426] Key type ._llcrypt unregistered [ 1024.304105] Key type ._llcrypt registered [ 1024.305210] Key type .llcrypt registered [ 1024.457117] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1024.465773] alg: No test for adler32 (adler32-zlib) [ 1025.408950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1025.469845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1025.507555] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 1026.465671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1026.481687] LNet: 50180:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1026.485820] LNetError: 50180:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1026.505744] LNet: Removed LNI 192.168.202.35@tcp [ 1026.528159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1026.644242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 1026.702138] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 1034.014498] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 13:18:21 (1790097501) [ 1034.124478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1034.151652] LNet: 50765:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1034.158943] LNetError: 50765:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1034.179408] LNet: Removed LNI 192.168.202.35@tcp [ 1034.212414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1034.323775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 1041.871685] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 13:18:28 (1790097508) [ 1043.261441] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 1044.923209] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 13:18:32 (1790097512) [ 1046.650361] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 1048.310463] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 13:18:35 (1790097515) [ 1049.751399] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 1051.515219] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 13:18:39 (1790097519) [ 1053.023618] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 1054.491795] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 13:18:42 (1790097522) [ 1056.461461] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 1058.238169] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing ========================================================== 13:18:45 (1790097525) [ 1058.353878] LNet: 52789:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1059.248191] Key type .llcrypt unregistered [ 1059.251519] Key type ._llcrypt unregistered [ 1066.750413] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1067.039399] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1067.041335] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1067.056391] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1069.668992] Key type ._llcrypt registered [ 1069.676912] Key type .llcrypt registered [ 1069.869167] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1069.898261] alg: No test for adler32 (adler32-zlib) [ 1070.858423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1070.976887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1071.017801] LNet: Added LNI 192.168.202.35@tcp [8/512/0/180] [ 1071.025328] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 1071.027198] LNet: 53111:0:(api-ni.c:2991:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1071.151333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1071.184235] LNet: 53131:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1071.195787] LNetError: 53131:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1071.203754] LNet: Removed LNI 10.1.2.3@tcp [ 1071.206253] LNetError: 53131:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1071.225039] LNet: Removed LNI 192.168.202.35@tcp [ 1071.273364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1071.370934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1071.397882] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 1071.484264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1071.519861] LNet: 53155:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1071.538134] LNetError: 53155:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1071.568596] LNet: Removed LNI 192.168.202.35@tcp [ 1071.625714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1071.724664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1071.768789] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 1071.773913] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1071.780169] LNet: 53163:0:(api-ni.c:2991:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1071.912398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1071.933128] LNet: 53181:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1071.942453] LNetError: 53181:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1071.950768] LNet: Removed LNI 10.1.2.3@tcp [ 1071.956772] LNetError: 53181:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1071.969938] LNet: Removed LNI 192.168.202.35@tcp [ 1071.989652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1072.058840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1072.087413] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 1072.091131] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1072.096581] LNet: 53189:0:(api-ni.c:2991:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1072.299449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1072.334384] LNet: 53215:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1072.340277] LNetError: 53215:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1072.358528] LNet: Removed LNI 10.1.2.3@tcp [ 1072.359976] LNetError: 53215:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1072.376937] LNet: Removed LNI 192.168.202.35@tcp [ 1072.411208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1072.483790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1072.508474] LNet: Added LNI 192.168.202.35@tcp [8/256/0/182] [ 1072.513156] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 1072.631851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1072.647970] LNet: 53243:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1072.659704] LNetError: 53243:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1072.670413] LNet: Removed LNI 10.1.2.3@tcp [ 1072.678597] LNetError: 53243:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1072.703174] LNet: Removed LNI 192.168.202.35@tcp [ 1072.735417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1072.833365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1072.866944] LNet: Added LNI 192.168.202.35@tcp [8/256/0/182] [ 1072.874988] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 1072.877899] LNet: 53251:0:(api-ni.c:2991:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1072.979869] LNet: Removed LNI 10.1.2.3@tcp [ 1072.993391] LNet: Removed LNI 192.168.202.35@tcp [ 1073.723282] Key type .llcrypt unregistered [ 1073.727559] Key type ._llcrypt unregistered [ 1085.492680] Key type ._llcrypt registered [ 1085.496492] Key type .llcrypt registered [ 1086.673776] Lustre: DEBUG MARKER: == sanity-lnet test 180: FastReg map error should not leak descriptors ========================================================== 13:19:14 (1790097554) [ 1088.607507] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_180 Need o2ib NETTYPE [ 1089.918179] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 13:19:17 (1790097557) [ 1090.378699] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1090.406520] alg: No test for adler32 (adler32-zlib) [ 1091.338672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1091.382896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1091.423335] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 1092.122369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.202.35 [ 1092.134831] LNet: 54516:0:(config.c:1740:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1092.142148] LNet: Removed LNI 192.168.202.35@tcp [ 1097.558286] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 13:19:25 (1790097565) [ 1097.842378] LNet: 55086:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1097.854878] LNetError: 55086:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1097.873129] LNet: Removed LNI 192.168.202.35@tcp [ 1098.450289] Key type .llcrypt unregistered [ 1098.452258] Key type ._llcrypt unregistered [ 1105.941544] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1106.470909] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1108.389830] Key type ._llcrypt registered [ 1108.396364] Key type .llcrypt registered [ 1108.738404] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1108.782646] alg: No test for adler32 (adler32-zlib) [ 1110.593579] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1119.683593] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 13:19:46 (1790097586) [ 1119.842571] LNet: 55984:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1119.860414] LNetError: 55984:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1119.904734] LNet: Removed LNI 10.1.2.3@tcp [ 1120.708496] Key type .llcrypt unregistered [ 1120.711864] Key type ._llcrypt unregistered [ 1127.584413] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1127.997511] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1128.000514] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1128.003774] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1129.659474] Key type ._llcrypt registered [ 1129.663692] Key type .llcrypt registered [ 1129.938879] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1129.965811] alg: No test for adler32 (adler32-zlib) [ 1130.977033] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1138.101193] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 13:20:05 (1790097605) [ 1140.010614] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 1141.944383] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 13:20:09 (1790097609) [ 1142.040058] LNet: 57162:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1142.049156] LNetError: 57162:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1142.056722] LNet: Removed LNI 10.1.2.3@tcp [ 1142.917770] Key type .llcrypt unregistered [ 1142.921189] Key type ._llcrypt unregistered [ 1151.061061] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1151.460401] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1151.474851] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1151.479912] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1153.195941] Key type ._llcrypt registered [ 1153.198835] Key type .llcrypt registered [ 1153.454729] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1153.481990] alg: No test for adler32 (adler32-zlib) [ 1154.474580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1154.608073] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1162.991473] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 13:20:30 (1790097630) [ 1163.507505] LNet: 58391:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1163.513156] LNetError: 58391:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1163.531915] LNet: Removed LNI 10.1.2.3@tcp [ 1164.281906] Key type .llcrypt unregistered [ 1164.283576] Key type ._llcrypt unregistered [ 1171.012198] Key type ._llcrypt registered [ 1171.018546] Key type .llcrypt registered [ 1171.341418] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1171.382569] alg: No test for adler32 (adler32-zlib) [ 1172.332464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1172.349123] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 1181.601197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.135@tcp [ 1182.341403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp -d 192.168.202.135@tcp -m GET -e local_interrupt -r 1 [ 1182.379605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp -d 192.168.202.35@tcp -m GET -e local_interrupt -r 1 [ 1182.429649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.135@tcp [ 1182.454843] LNet: There was an unexpected network error while writing to 192.168.202.135: rc = -22 [ 1182.655808] LNet: 1 local NIs in recovery (showing 1): 192.168.202.35@tcp [ 1182.748583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1182.802236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1182.880158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1182.952618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1183.016651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1183.067375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1183.109168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1183.164249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1183.240431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1183.277573] LNet: 58831:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1183.356710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1183.453194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1183.506936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1183.575810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1183.638184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1183.683551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1183.728804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1184.495541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp -d 192.168.202.135@tcp -m GET -e local_dropped -r 1 [ 1184.529325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp -d 192.168.202.35@tcp -m GET -e local_dropped -r 1 [ 1184.607863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.135@tcp [ 1184.638472] LNet: There was an unexpected network error while writing to 192.168.202.135: rc = -22 [ 1184.880835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1184.932133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1184.960734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1184.990370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1185.027977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1185.079234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1185.130775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1185.183735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1185.227641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1185.246659] LNet: 58943:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1185.276803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1185.342759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1185.400902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1185.440745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1185.482946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1185.534272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1185.594968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1186.232237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp -d 192.168.202.135@tcp -m GET -e local_aborted -r 1 [ 1186.263465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp -d 192.168.202.35@tcp -m GET -e local_aborted -r 1 [ 1186.286749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.135@tcp [ 1186.310152] LNet: There was an unexpected network error while writing to 192.168.202.135: rc = -22 [ 1186.526379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1186.549308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1186.578221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1186.603908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1186.631356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1186.645622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1186.659194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1186.672188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1186.685465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1186.694078] LNet: 59054:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1186.725817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1186.768950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1186.806551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1186.853687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1186.897984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1186.923099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1186.944359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1187.563429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp -d 192.168.202.135@tcp -m GET -e local_no_route -r 1 [ 1187.592402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp -d 192.168.202.35@tcp -m GET -e local_no_route -r 1 [ 1187.642894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.135@tcp [ 1187.845551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1187.869272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1187.893262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1187.927612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1187.986316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1188.050753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1188.088467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1188.110839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1188.144756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1188.204168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1188.304748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1188.355482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1188.418961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1188.475417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1188.563251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1188.643048] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1189.523172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp -d 192.168.202.135@tcp -m GET -e local_timeout -r 1 [ 1189.556867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp -d 192.168.202.35@tcp -m GET -e local_timeout -r 1 [ 1189.610796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.135@tcp [ 1189.631932] LNet: There was an unexpected network error while writing to 192.168.202.135: rc = -22 [ 1189.639467] LNet: Skipped 2 previous similar messages [ 1189.929630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1189.961665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1189.988367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1190.034469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1190.072029] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1190.104861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1190.143838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1190.251401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1190.330648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1190.383281] LNet: 59277:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1190.397568] LNet: 59277:0:(api-ni.c:301:recovery_interval_set()) Skipped 1 previous similar message [ 1190.438590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1190.524222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1190.577663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1190.625553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1190.704669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1190.773698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1190.814569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1191.557482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp -d 192.168.202.135@tcp -m GET -e local_error -r 1 [ 1191.613286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp -d 192.168.202.35@tcp -m GET -e local_error -r 1 [ 1191.656274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.135@tcp [ 1192.066491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1192.145919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1192.271394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1192.360159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1192.415408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1192.454687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1192.492207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1192.533040] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1192.567605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1192.608525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1192.642615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1192.676582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1192.704388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1192.737846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1192.773311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1192.811873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1193.143790] LNet: 59429:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1193.148982] LNetError: 59429:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1194.213822] LNet: Removed LNI 192.168.202.35@tcp [ 1194.981577] Key type .llcrypt unregistered [ 1194.992789] Key type ._llcrypt unregistered [ 1215.991747] Key type ._llcrypt registered [ 1215.996201] Key type .llcrypt registered [ 1217.762926] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 13:21:24 (1790097684) [ 1218.662396] Key type .llcrypt unregistered [ 1218.664038] Key type ._llcrypt unregistered [ 1227.326647] Key type ._llcrypt registered [ 1227.330834] Key type .llcrypt registered [ 1227.643488] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1227.669146] alg: No test for adler32 (adler32-zlib) [ 1228.664695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1228.725409] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 1237.963121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.135@tcp [ 1249.315747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1249.363408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1249.387617] LNet: Added LNI 192.168.202.35@tcp1 [8/256/0/180] [ 1250.942928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp -d 192.168.202.135@tcp -m GET -e local_interrupt -r 1 [ 1251.018301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp -d 192.168.202.135@tcp1 -m GET -e local_interrupt -r 1 [ 1251.057486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp -d 192.168.202.35@tcp -m GET -e local_interrupt -r 1 [ 1251.092116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp -d 192.168.202.35@tcp1 -m GET -e local_interrupt -r 1 [ 1251.141833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp1 -d 192.168.202.135@tcp -m GET -e local_interrupt -r 1 [ 1251.178401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp1 -d 192.168.202.135@tcp1 -m GET -e local_interrupt -r 1 [ 1251.216263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp1 -d 192.168.202.35@tcp -m GET -e local_interrupt -r 1 [ 1251.255097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp1 -d 192.168.202.35@tcp1 -m GET -e local_interrupt -r 1 [ 1251.286642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.135@tcp [ 1251.305973] LNet: There was an unexpected network error while writing to 192.168.202.135: rc = -22 [ 1251.314500] LNet: 1 local NIs in recovery (showing 1): 192.168.202.35@tcp [ 1251.745607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1251.840142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1251.906912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1251.962626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1252.006927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1252.050238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1252.092740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1252.137575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1252.170388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1252.181191] LNet: 60902:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1252.202196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1252.234086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1252.256463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1252.281123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1252.312353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1252.332280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1252.353094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1253.048333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp -d 192.168.202.135@tcp -m GET -e local_dropped -r 1 [ 1253.093496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp -d 192.168.202.135@tcp1 -m GET -e local_dropped -r 1 [ 1253.159879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp -d 192.168.202.35@tcp -m GET -e local_dropped -r 1 [ 1253.196082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp -d 192.168.202.35@tcp1 -m GET -e local_dropped -r 1 [ 1253.286234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp1 -d 192.168.202.135@tcp -m GET -e local_dropped -r 1 [ 1253.351518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp1 -d 192.168.202.135@tcp1 -m GET -e local_dropped -r 1 [ 1253.391611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp1 -d 192.168.202.35@tcp -m GET -e local_dropped -r 1 [ 1253.428055] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp1 -d 192.168.202.35@tcp1 -m GET -e local_dropped -r 1 [ 1253.455226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.135@tcp [ 1253.473509] LNet: There was an unexpected network error while writing to 192.168.202.135: rc = -22 [ 1253.478486] LNet: Skipped 2 previous similar messages [ 1253.691528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1253.755204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1253.805944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1253.860733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1253.948770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1254.043310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1254.098830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1254.132198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1254.199351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1254.237319] LNet: 61025:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1254.265429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1254.328954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1254.388690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1254.466856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1254.503724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1254.521975] LNet: There was an unexpected network error while writing to 192.168.202.35: rc = -22 [ 1254.529940] LNet: Skipped 3 previous similar messages [ 1254.563905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1254.609141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1255.288945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp -d 192.168.202.135@tcp -m GET -e local_aborted -r 1 [ 1255.329639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp -d 192.168.202.135@tcp1 -m GET -e local_aborted -r 1 [ 1255.383684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp -d 192.168.202.35@tcp -m GET -e local_aborted -r 1 [ 1255.435414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp -d 192.168.202.35@tcp1 -m GET -e local_aborted -r 1 [ 1255.499864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp1 -d 192.168.202.135@tcp -m GET -e local_aborted -r 1 [ 1255.544782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp1 -d 192.168.202.135@tcp1 -m GET -e local_aborted -r 1 [ 1255.604870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp1 -d 192.168.202.35@tcp -m GET -e local_aborted -r 1 [ 1255.676462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp1 -d 192.168.202.35@tcp1 -m GET -e local_aborted -r 1 [ 1255.730256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.135@tcp [ 1256.168228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1256.211080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1256.271809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1256.345859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1256.415306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1256.455822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1256.524967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1256.574686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1256.618587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1256.635690] LNet: 61149:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1256.652476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1256.705760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1256.746662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1256.788746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1256.817657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1256.843938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1256.874397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1257.521217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp -d 192.168.202.135@tcp -m GET -e local_no_route -r 1 [ 1257.584889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp -d 192.168.202.135@tcp1 -m GET -e local_no_route -r 1 [ 1257.665688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp -d 192.168.202.35@tcp -m GET -e local_no_route -r 1 [ 1257.706439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp -d 192.168.202.35@tcp1 -m GET -e local_no_route -r 1 [ 1257.752583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp1 -d 192.168.202.135@tcp -m GET -e local_no_route -r 1 [ 1257.814943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp1 -d 192.168.202.135@tcp1 -m GET -e local_no_route -r 1 [ 1257.857886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp1 -d 192.168.202.35@tcp -m GET -e local_no_route -r 1 [ 1257.901581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp1 -d 192.168.202.35@tcp1 -m GET -e local_no_route -r 1 [ 1257.972810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.135@tcp [ 1258.004609] LNet: There was an unexpected network error while writing to 192.168.202.135: rc = -22 [ 1258.012297] LNet: Skipped 3 previous similar messages [ 1258.280320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1258.319417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1258.351199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1258.391944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1258.432395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1258.461624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1258.491466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1258.518305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1258.552544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1258.599419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1258.664362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1258.705081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1258.794606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1258.832364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1258.866295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1258.903917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1259.497555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp -d 192.168.202.135@tcp -m GET -e local_timeout -r 1 [ 1259.543337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp -d 192.168.202.135@tcp1 -m GET -e local_timeout -r 1 [ 1259.609536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp -d 192.168.202.35@tcp -m GET -e local_timeout -r 1 [ 1259.683713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp -d 192.168.202.35@tcp1 -m GET -e local_timeout -r 1 [ 1259.730450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp1 -d 192.168.202.135@tcp -m GET -e local_timeout -r 1 [ 1259.771358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp1 -d 192.168.202.135@tcp1 -m GET -e local_timeout -r 1 [ 1259.828209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp1 -d 192.168.202.35@tcp -m GET -e local_timeout -r 1 [ 1259.877948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp1 -d 192.168.202.35@tcp1 -m GET -e local_timeout -r 1 [ 1259.939610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.135@tcp [ 1260.329318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1260.355143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1260.397283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1260.455404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1260.493305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1260.535488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1260.589531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1260.640542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1260.670958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1260.697160] LNet: 61395:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1260.716894] LNet: 61395:0:(api-ni.c:301:recovery_interval_set()) Skipped 1 previous similar message [ 1260.750906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1260.808079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1260.864460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1260.905576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1260.940396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1260.995244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1261.043681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1261.750068] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp -d 192.168.202.135@tcp -m GET -e local_error -r 1 [ 1261.780365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp -d 192.168.202.135@tcp1 -m GET -e local_error -r 1 [ 1261.806814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp -d 192.168.202.35@tcp -m GET -e local_error -r 1 [ 1261.830567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp -d 192.168.202.35@tcp1 -m GET -e local_error -r 1 [ 1261.867065] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp1 -d 192.168.202.135@tcp -m GET -e local_error -r 1 [ 1261.911624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp1 -d 192.168.202.135@tcp1 -m GET -e local_error -r 1 [ 1261.972884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp1 -d 192.168.202.35@tcp -m GET -e local_error -r 1 [ 1262.016976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp1 -d 192.168.202.35@tcp1 -m GET -e local_error -r 1 [ 1262.050334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.135@tcp [ 1262.068680] LNet: There was an unexpected network error while writing to 192.168.202.135: rc = -22 [ 1262.075978] LNet: Skipped 5 previous similar messages [ 1262.302160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1262.337567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1262.394856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1262.435623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1262.481561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1262.522890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1262.574453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1262.619418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1262.670466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1262.702514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1262.746530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1262.810842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1262.857898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1262.909989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1262.959513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1263.005253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1269.416614] LNet: 61595:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1269.434921] LNetError: 61595:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1269.449953] LNet: Removed LNI 192.168.202.35@tcp [ 1270.503168] LNet: Removed LNI 192.168.202.35@tcp1 [ 1270.802841] Key type .llcrypt unregistered [ 1270.806088] Key type ._llcrypt unregistered [ 1290.478879] Key type ._llcrypt registered [ 1290.484867] Key type .llcrypt registered [ 1292.026849] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 13:22:39 (1790097759) [ 1293.423635] Key type .llcrypt unregistered [ 1293.425529] Key type ._llcrypt unregistered [ 1302.258469] Key type ._llcrypt registered [ 1302.260905] Key type .llcrypt registered [ 1302.417894] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1302.428845] alg: No test for adler32 (adler32-zlib) [ 1303.368706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1303.397427] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 1310.485294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.135@tcp [ 1311.131666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp -d 192.168.202.135@tcp -m GET -e remote_dropped -r 1 [ 1311.178947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp -d 192.168.202.35@tcp -m GET -e remote_dropped -r 1 [ 1311.243692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.135@tcp [ 1311.264433] LNet: There was an unexpected network error while writing to 192.168.202.135: rc = -22 [ 1311.638511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1311.673893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1311.704251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1311.734469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1311.760887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1311.791690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1311.823378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1311.869811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1311.913761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1311.974906] LNet: 62847:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1312.005847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1312.085963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1312.147951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1312.203808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1312.254418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1312.303923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1312.351887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1313.011884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp -d 192.168.202.135@tcp -m GET -e remote_error -r 1 [ 1313.056802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp -d 192.168.202.35@tcp -m GET -e remote_error -r 1 [ 1313.121061] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.135@tcp [ 1313.166947] LNet: There was an unexpected network error while writing to 192.168.202.135: rc = -22 [ 1313.528980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1313.588174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1313.648603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1313.688502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1313.716465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1313.764433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1313.791447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1313.830223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1313.859766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1313.889629] LNet: 62959:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1313.937844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1313.990926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1314.038940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1314.103582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1314.176500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1314.305252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1314.372488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1315.167544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp -d 192.168.202.135@tcp -m GET -e remote_timeout -r 1 [ 1315.200655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp -d 192.168.202.35@tcp -m GET -e remote_timeout -r 1 [ 1315.229096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.135@tcp [ 1315.249767] LNet: There was an unexpected network error while writing to 192.168.202.135: rc = -22 [ 1315.553501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1315.580249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1315.599329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1315.623726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1315.646665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1315.674517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1315.694701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1315.715852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1315.739394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1315.751816] LNet: 63070:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1315.769889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1315.794055] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1315.820318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1315.856163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1315.882117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1315.905934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1315.959977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1316.271048] LNet: 63111:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1316.284376] LNetError: 63111:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1316.312631] LNet: Removed LNI 192.168.202.35@tcp [ 1316.604562] Key type .llcrypt unregistered [ 1316.608193] Key type ._llcrypt unregistered [ 1334.747070] Key type ._llcrypt registered [ 1334.748739] Key type .llcrypt registered [ 1336.011749] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 13:23:23 (1790097803) [ 1336.910335] Key type .llcrypt unregistered [ 1336.914036] Key type ._llcrypt unregistered [ 1343.512272] Key type ._llcrypt registered [ 1343.527356] Key type .llcrypt registered [ 1343.835657] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1343.864910] alg: No test for adler32 (adler32-zlib) [ 1344.777533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1344.806113] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 1352.429950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.135@tcp [ 1361.844864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1361.885220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1361.905254] LNet: Added LNI 192.168.202.35@tcp1 [8/256/0/180] [ 1363.329077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp -d 192.168.202.135@tcp -m GET -e remote_dropped -r 1 [ 1363.356227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp -d 192.168.202.135@tcp1 -m GET -e remote_dropped -r 1 [ 1363.411473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp -d 192.168.202.35@tcp -m GET -e remote_dropped -r 1 [ 1363.443324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp -d 192.168.202.35@tcp1 -m GET -e remote_dropped -r 1 [ 1363.473219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp1 -d 192.168.202.135@tcp -m GET -e remote_dropped -r 1 [ 1363.522486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp1 -d 192.168.202.135@tcp1 -m GET -e remote_dropped -r 1 [ 1363.585129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp1 -d 192.168.202.35@tcp -m GET -e remote_dropped -r 1 [ 1363.633062] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp1 -d 192.168.202.35@tcp1 -m GET -e remote_dropped -r 1 [ 1363.672931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.135@tcp [ 1363.695283] LNet: There was an unexpected network error while writing to 192.168.202.135: rc = -22 [ 1363.700978] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.135@tcp [ 1363.896628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1363.928932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1363.961878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1364.004268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1364.048527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1364.081815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1364.130375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1364.236089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1364.279783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1364.299169] LNet: 64581:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1364.320872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1364.364813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1364.397640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1364.440155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1364.471675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1364.506594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1364.536627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1364.585170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1365.171196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp -d 192.168.202.135@tcp -m GET -e remote_error -r 1 [ 1365.227381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp -d 192.168.202.135@tcp1 -m GET -e remote_error -r 1 [ 1365.287145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp -d 192.168.202.35@tcp -m GET -e remote_error -r 1 [ 1365.333268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp -d 192.168.202.35@tcp1 -m GET -e remote_error -r 1 [ 1365.375331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp1 -d 192.168.202.135@tcp -m GET -e remote_error -r 1 [ 1365.424357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp1 -d 192.168.202.135@tcp1 -m GET -e remote_error -r 1 [ 1365.478371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp1 -d 192.168.202.35@tcp -m GET -e remote_error -r 1 [ 1365.517369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp1 -d 192.168.202.35@tcp1 -m GET -e remote_error -r 1 [ 1365.560271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.135@tcp [ 1365.578510] LNet: There was an unexpected network error while writing to 192.168.202.135: rc = -22 [ 1365.585873] LNet: Skipped 2 previous similar messages [ 1365.927539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1365.967239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1366.023057] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1366.061460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1366.109796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1366.174419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1366.218281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1366.262088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1366.342419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1366.379518] LNet: 64706:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1366.410366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1366.454932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1366.506596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1366.534607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1366.563179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1366.588271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1366.611626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1366.641991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1367.034497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp -d 192.168.202.135@tcp -m GET -e remote_timeout -r 1 [ 1367.058393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp -d 192.168.202.135@tcp1 -m GET -e remote_timeout -r 1 [ 1367.086437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp -d 192.168.202.35@tcp -m GET -e remote_timeout -r 1 [ 1367.118344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp -d 192.168.202.35@tcp1 -m GET -e remote_timeout -r 1 [ 1367.144638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp1 -d 192.168.202.135@tcp -m GET -e remote_timeout -r 1 [ 1367.179966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp1 -d 192.168.202.135@tcp1 -m GET -e remote_timeout -r 1 [ 1367.203850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp1 -d 192.168.202.35@tcp -m GET -e remote_timeout -r 1 [ 1367.237410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp1 -d 192.168.202.35@tcp1 -m GET -e remote_timeout -r 1 [ 1367.270383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.135@tcp [ 1367.287102] LNet: There was an unexpected network error while writing to 192.168.202.135: rc = -22 [ 1367.528193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1367.549784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1367.572387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1367.593814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1367.625414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1367.668604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1367.708389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1367.734906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1367.759809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1367.773633] LNet: 64831:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1367.786176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1367.810399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1367.834572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1367.854868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1367.887629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1367.918972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1367.949265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1367.988119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1372.968422] LNet: 64909:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1372.980896] LNetError: 64909:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1374.048250] LNet: Removed LNI 192.168.202.35@tcp [ 1374.050321] LNetError: 64909:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1374.083613] LNet: Removed LNI 192.168.202.35@tcp1 [ 1374.552878] Key type .llcrypt unregistered [ 1374.557539] Key type ._llcrypt unregistered [ 1392.211659] Key type ._llcrypt registered [ 1392.216059] Key type .llcrypt registered [ 1393.920501] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 13:24:20 (1790097860) [ 1395.613552] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 1397.629197] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 13:24:24 (1790097864) [ 1398.456392] Key type .llcrypt unregistered [ 1398.458465] Key type ._llcrypt unregistered [ 1405.311239] Key type ._llcrypt registered [ 1405.313586] Key type .llcrypt registered [ 1405.457605] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1405.470622] alg: No test for adler32 (adler32-zlib) [ 1406.372234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1406.394275] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 1414.559925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.135@tcp [ 1415.227486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp -d 192.168.202.135@tcp -m GET -e network_timeout -r 1 [ 1415.255481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp -d 192.168.202.35@tcp -m GET -e network_timeout -r 1 [ 1415.288817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.135@tcp [ 1415.307106] LNet: There was an unexpected network error while writing to 192.168.202.135: rc = -22 [ 1415.600203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1415.652399] LNet: 1 local NIs in recovery (showing 1): 192.168.202.35@tcp [ 1415.675001] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1415.739958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1415.801726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1415.837815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1415.900118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1415.944848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1416.005876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1416.058934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1416.088792] LNet: 66456:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1416.106972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1416.152620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1416.175604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1416.228386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1416.312820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1416.361846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1416.677436] LNet: 66495:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1416.681857] LNetError: 66495:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1416.693178] LNet: Removed LNI 192.168.202.35@tcp [ 1417.195718] Key type .llcrypt unregistered [ 1417.199849] Key type ._llcrypt unregistered [ 1436.807623] Key type ._llcrypt registered [ 1436.809631] Key type .llcrypt registered [ 1437.067702] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1437.077619] alg: No test for adler32 (adler32-zlib) [ 1438.056962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1438.086296] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 1446.752936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.135@tcp [ 1456.843865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1456.909357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1456.943287] LNet: Added LNI 192.168.202.35@tcp1 [8/256/0/180] [ 1458.342482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp -d 192.168.202.135@tcp -m GET -e network_timeout -r 1 [ 1458.386590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp -d 192.168.202.135@tcp1 -m GET -e network_timeout -r 1 [ 1458.431507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp -d 192.168.202.35@tcp -m GET -e network_timeout -r 1 [ 1458.456565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp -d 192.168.202.35@tcp1 -m GET -e network_timeout -r 1 [ 1458.484988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp1 -d 192.168.202.135@tcp -m GET -e network_timeout -r 1 [ 1458.514279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp1 -d 192.168.202.135@tcp1 -m GET -e network_timeout -r 1 [ 1458.540958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp1 -d 192.168.202.35@tcp -m GET -e network_timeout -r 1 [ 1458.569531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp1 -d 192.168.202.35@tcp1 -m GET -e network_timeout -r 1 [ 1458.607225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.135@tcp [ 1458.626449] LNet: There was an unexpected network error while writing to 192.168.202.135: rc = -22 [ 1458.833661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1458.857561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1458.878721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1458.902697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1458.924622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1458.971463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1459.014277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1459.042644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1459.067624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1459.079950] LNet: 67377:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1459.101613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1459.140184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1459.173643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1459.209542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1459.241176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1459.275319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1463.836499] LNet: There was an unexpected network error while writing to 192.168.202.135: rc = -22 [ 1464.256970] LNet: 67451:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1464.264600] LNetError: 67451:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1464.274346] LNet: Removed LNI 192.168.202.35@tcp [ 1464.288609] LNet: Removed LNI 192.168.202.35@tcp1 [ 1464.676992] Key type .llcrypt unregistered [ 1464.678111] Key type ._llcrypt unregistered [ 1482.448642] Key type ._llcrypt registered [ 1482.451117] Key type .llcrypt registered [ 1483.852923] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 13:25:51 (1790097951) [ 1484.224414] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1484.244843] alg: No test for adler32 (adler32-zlib) [ 1485.232933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1485.301787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1485.336914] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 1486.072052] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1486.107440] LNet: Added LNI 192.168.202.35@tcp1 [8/256/0/180] [ 1486.813339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.35@tcp [ 1486.872247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1486.918967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1486.996337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1487.063413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1487.117369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1487.145334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.202.35@tcp [ 1487.166884] LNetError: 68552:0:(api-ni.c:4478:lnet_ni_set_healthv()) manually adding local NI 192.168.202.35@tcp to recovery [ 1487.327431] LNet: 1 local NIs in recovery (showing 1): 192.168.202.35@tcp [ 1487.330307] LNet: There was an unexpected network error while writing to 192.168.202.35: rc = -22 [ 1489.375513] LNet: There was an unexpected network error while writing to 192.168.202.35: rc = -22 [ 1489.646424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1493.471657] LNet: There was an unexpected network error while writing to 192.168.202.35: rc = -22 [ 1494.315289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1494.349866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1494.414697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1494.437373] LNet: 68659:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1494.443229] LNetError: 68659:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1495.455180] LNet: Removed LNI 192.168.202.35@tcp [ 1495.463401] LNetError: 68659:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1495.474613] LNet: Removed LNI 192.168.202.35@tcp1 [ 1495.517954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1495.640317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1495.672666] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 1496.462839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1496.488811] LNet: Added LNI 192.168.202.35@tcp1 [8/256/0/180] [ 1497.396057] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.35@tcp [ 1497.439736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1497.511381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1497.559438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1497.609365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1497.662989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1497.695919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1497.726646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.202.35@tcp [ 1497.743761] LNetError: 68932:0:(api-ni.c:4478:lnet_ni_set_healthv()) manually adding local NI 192.168.202.35@tcp to recovery [ 1498.656657] LNet: There was an unexpected network error while writing to 192.168.202.35: rc = -22 [ 1501.328468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1504.799428] LNet: There was an unexpected network error while writing to 192.168.202.35: rc = -22 [ 1504.806316] LNet: Skipped 1 previous similar message [ 1509.350488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1509.383447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1509.410405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1515.774788] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 13:26:23 (1790097983) [ 1515.994479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1516.032214] LNet: 69650:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1516.041676] LNetError: 69650:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1517.087103] LNet: Removed LNI 192.168.202.35@tcp [ 1517.099869] LNet: Removed LNI 192.168.202.35@tcp1 [ 1517.147533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1517.239916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1517.277337] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 1517.982788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1517.998385] LNet: Added LNI 192.168.202.35@tcp1 [8/256/0/180] [ 1519.125516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.35@tcp [ 1519.280177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1519.316245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1519.353471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1519.395161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1519.427554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1519.463659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.202.35@tcp --health 0 [ 1519.571763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1520.287443] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.35@tcp [ 1520.288400] LNet: There was an unexpected network error while writing to 192.168.202.35: rc = -22 [ 1520.315983] LNet: Skipped 1 previous similar message [ 1535.004883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1535.164675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1535.224877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1535.275578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.202.35@tcp --health 500 [ 1535.403557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1542.252902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1542.332794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1542.366824] LNet: 70143:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1542.379023] LNetError: 70143:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1542.388520] LNetError: 70143:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1542.400718] LNet: Removed LNI 192.168.202.35@tcp [ 1542.425875] LNet: Removed LNI 192.168.202.35@tcp1 [ 1542.473141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1542.585202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1542.631606] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 1543.613550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1543.653411] LNet: Added LNI 192.168.202.35@tcp1 [8/256/0/180] [ 1544.696918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.35@tcp [ 1544.727933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1544.797454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1544.832601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1544.862483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1544.900524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1544.927901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1544.952778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.202.35@tcp --health 0 [ 1545.695379] LNet: There was an unexpected network error while writing to 192.168.202.35: rc = -22 [ 1545.699743] LNet: Skipped 3 previous similar messages [ 1546.265075] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1556.726560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1556.758385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1556.791061] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1562.953720] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 13:27:10 (1790098030) [ 1563.520493] LNet: 71189:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1563.528694] LNetError: 71189:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1563.542892] LNetError: 71189:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1563.553193] LNet: Removed LNI 192.168.202.35@tcp [ 1563.564515] LNet: Removed LNI 192.168.202.35@tcp1 [ 1564.247956] Key type .llcrypt unregistered [ 1564.251179] Key type ._llcrypt unregistered [ 1570.188738] Key type ._llcrypt registered [ 1570.189916] Key type .llcrypt registered [ 1570.404471] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1570.416284] alg: No test for adler32 (adler32-zlib) [ 1571.358545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1571.405822] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 1580.727427] Lustre: DEBUG MARKER: Initial discovery [ 1580.850842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.202.135@tcp [ 1583.005402] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1583.260315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1583.322242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.202.135@tcp [ 1583.345702] LNet: There was an unexpected network error while writing to 192.168.202.135: rc = -22 [ 1583.363510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1583.418561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.202.135@tcp -r 1 -m GET -l 3 [ 1583.465231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.202.135@tcp -r 1 -m GET -e local_error [ 1583.498635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.202.135@tcp -r 1 -m PUT -l 6 [ 1583.711552] LNet: 1 local NIs in recovery (showing 1): 192.168.202.35@tcp [ 1585.114786] Lustre: DEBUG MARKER: Force oleg235-client.virtnet to discover 192.168.202.135@tcp (in background) [ 1585.265672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.202.135@tcp [ 1587.019803] Lustre: DEBUG MARKER: Force oleg235-server to discover 192.168.202.35@tcp [ 1588.384859] LNet: There was an unexpected network error while writing to 192.168.202.135: rc = -22 [ 1589.540991] Lustre: DEBUG MARKER: Wait for 71944 [ 1591.864830] Lustre: DEBUG MARKER: Finished wait on 71944 [ 1591.921771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay del -a [ 1592.005034] LNet: There was an unexpected network error while writing to 192.168.202.135: rc = -22 [ 1592.007465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1592.522617] LNet: 72401:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1592.529530] LNetError: 72401:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1592.551720] LNet: Removed LNI 192.168.202.35@tcp [ 1593.275924] Key type .llcrypt unregistered [ 1593.281068] Key type ._llcrypt unregistered [ 1616.587749] Key type ._llcrypt registered [ 1616.597364] Key type .llcrypt registered [ 1618.462079] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 13:28:05 (1790098085) [ 1619.182567] Key type .llcrypt unregistered [ 1619.184800] Key type ._llcrypt unregistered [ 1626.731020] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1627.027486] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1627.035752] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1627.046856] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1629.889245] Key type ._llcrypt registered [ 1629.891772] Key type .llcrypt registered [ 1630.132537] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1630.145338] alg: No test for adler32 (adler32-zlib) [ 1631.192630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1631.281177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1631.352649] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 1632.078619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1632.107484] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1639.310228] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 13:28:26 (1790098106) [ 1639.424962] LNet: 74470:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1639.429171] LNetError: 74470:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1639.437827] LNet: Removed LNI 10.1.2.3@tcp [ 1639.449613] LNet: Removed LNI 192.168.202.35@tcp [ 1640.343549] Key type .llcrypt unregistered [ 1640.346544] Key type ._llcrypt unregistered [ 1648.411895] Key type ._llcrypt registered [ 1648.413699] Key type .llcrypt registered [ 1648.631965] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1648.651689] alg: No test for adler32 (adler32-zlib) [ 1649.572929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1649.631626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1649.666039] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 1650.265375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1650.284875] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1651.160305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.202.35@tcp [ 1652.256394] LNetError: Unexpected error -22 connecting to 192.168.202.35@tcp at host 192.168.202.35:988 [ 1657.343172] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 13:28:44 (1790098124) [ 1657.738243] LNet: 75609:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1657.749284] LNetError: 75609:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1657.755408] LNet: Removed LNI 10.1.2.3@tcp [ 1657.771936] LNet: Removed LNI 192.168.202.35@tcp [ 1658.509653] Key type .llcrypt unregistered [ 1658.511798] Key type ._llcrypt unregistered [ 1666.289491] Key type ._llcrypt registered [ 1666.291849] Key type .llcrypt registered [ 1666.478054] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1666.493565] alg: No test for adler32 (adler32-zlib) [ 1667.378563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1667.443314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1667.464764] LNet: Added LNI 192.168.202.35@tcp1 [8/256/0/180] [ 1668.231864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1668.292446] LNet: Added LNI 192.168.202.35@tcp2 [8/256/0/180] [ 1669.357291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.202.35@tcp1 --nid 192.168.202.35@tcp2 [ 1673.647725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.35@tcp1 192.168.202.35@tcp2 [ 1673.672648] LNetError: 76444:0:(lib-move.c:2094:lnet_handle_find_routed_path()) Unable to send message from 192.168.202.35@tcp1 to 192.168.202.35@tcp2 - Route table may be misconfigured [ 1673.694912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.35@tcp2 192.168.202.35@tcp1 [ 1680.173941] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 13:29:07 (1790098147) [ 1680.504458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1680.542590] LNet: 77017:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1680.564163] LNetError: 77017:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1680.570763] LNet: Removed LNI 192.168.202.35@tcp1 [ 1681.653703] LNet: Removed LNI 192.168.202.35@tcp2 [ 1681.725567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1681.820490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1681.840939] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 1682.683134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1682.710906] LNet: Added LNI 192.168.202.35@tcp1 [8/256/0/180] [ 1683.712429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.35@tcp [ 1683.776536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.35@tcp 192.168.202.35@tcp [ 1683.812737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.35@tcp1 192.168.202.35@tcp1 [ 1683.869756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.202.35@tcp -d 192.168.202.35@tcp -e network_timeout [ 1683.929534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.202.35@tcp -d 192.168.202.35@tcp1 -e network_timeout [ 1683.979147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.202.35@tcp1 -d 192.168.202.35@tcp -e network_timeout [ 1684.015573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.202.35@tcp1 -d 192.168.202.35@tcp1 -e network_timeout [ 1684.046420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.35@tcp [ 1684.069079] LNet: There was an unexpected network error while writing to 192.168.202.35: rc = -22 [ 1684.085799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1684.159252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1684.260717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1684.833576] LNet: 1 local NIs in recovery (showing 1): 192.168.202.35@tcp [ 1690.755807] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 13:29:18 (1790098158) [ 1690.927333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1690.945385] LNet: 77878:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1690.955308] LNetError: 77878:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1690.965583] LNetError: 77878:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1690.972339] LNet: Removed LNI 192.168.202.35@tcp [ 1692.015975] LNet: Removed LNI 192.168.202.35@tcp1 [ 1692.084121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1692.158201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1692.457905] LNet: 77916:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1692.867427] Key type .llcrypt unregistered [ 1692.871383] Key type ._llcrypt unregistered [ 1706.333460] Key type ._llcrypt registered [ 1706.338578] Key type .llcrypt registered [ 1707.815543] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 13:29:34 (1790098174) [ 1708.433854] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1708.463820] alg: No test for adler32 (adler32-zlib) [ 1709.354180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1710.749428] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1712.738121] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 13:29:39 (1790098179) [ 1712.867327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1712.901466] LNet: 79006:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1712.944903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1713.013706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1713.052471] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 1713.773527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1713.788192] LNet: Added LNI 192.168.202.35@tcp1 [8/256/0/180] [ 1714.849141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.35@tcp [ 1714.879160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.35@tcp1 [ 1714.910164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.35@tcp1 [ 1721.978876] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 13:29:49 (1790098189) [ 1724.101085] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg235-server [ 1725.814600] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 13:29:53 (1790098193) [ 1727.860844] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg235-server [ 1729.632775] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 13:29:56 (1790098196) [ 1731.564540] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg235-server [ 1733.539790] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 13:30:00 (1790098200) [ 1735.728381] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg235-server [ 1737.933729] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 13:30:04 (1790098204) [ 1739.681450] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg235-server [ 1741.696577] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 13:30:08 (1790098208) [ 1744.123190] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg235-server [ 1745.929112] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 13:30:13 (1790098213) [ 1748.232916] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg235-server [ 1749.833583] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 13:30:17 (1790098217) [ 1751.880599] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg235-server [ 1753.670220] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 13:30:21 (1790098221) [ 1755.712155] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg235-server [ 1757.616955] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 13:30:24 (1790098224) [ 1757.801185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1757.813963] LNet: 82887:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1757.818589] LNetError: 82887:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1758.879377] LNet: Removed LNI 192.168.202.35@tcp [ 1758.912858] LNetError: 82887:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1758.934778] LNet: Removed LNI 192.168.202.35@tcp1 [ 1758.962817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1759.020765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1759.039490] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 1759.575095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1759.689795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.35@tcp [ 1759.903703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1759.931912] LNet: 83038:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1759.940611] LNetError: 83038:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1759.952926] LNet: Removed LNI 192.168.202.35@tcp [ 1759.985300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1760.058664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1760.106887] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 1760.871234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.35@tcp [ 1761.049970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1762.096103] LNet: Removed LNI 192.168.202.35@tcp [ 1762.143346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1762.236297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1762.267599] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 1763.153923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1763.265241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.35@tcp [ 1763.544108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1763.557833] LNet: 83331:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1763.561192] LNet: 83331:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1763.564350] LNetError: 83331:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1763.567616] LNetError: 83331:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1763.577715] LNet: Removed LNI 192.168.202.35@tcp [ 1763.596609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1763.658269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1763.703086] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 1764.441338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.35@tcp [ 1764.566523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1765.605631] LNet: Removed LNI 192.168.202.35@tcp [ 1765.646201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1765.721797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1765.765587] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 1766.562450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1766.677245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.35@tcp [ 1766.962959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1768.102580] LNet: Removed LNI 192.168.202.35@tcp [ 1768.145669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1768.224581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1768.274667] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 1769.201488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.35@tcp [ 1769.446618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1769.466619] LNet: 83768:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1769.473471] LNet: 83768:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 1769.489768] LNetError: 83768:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1769.495491] LNetError: 83768:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1769.517066] LNet: Removed LNI 192.168.202.35@tcp [ 1769.534664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1769.596846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1769.636246] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 1770.391075] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1770.483317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.35@tcp [ 1770.664500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1771.762715] LNet: Removed LNI 192.168.202.35@tcp [ 1771.783174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1771.876844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1771.912498] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 1772.769555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.35@tcp [ 1772.927546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1773.990544] LNet: Removed LNI 192.168.202.35@tcp [ 1774.025524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1774.084618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1774.120406] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 1774.747500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1774.813708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.35@tcp [ 1775.063498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1776.166828] LNet: Removed LNI 192.168.202.35@tcp [ 1776.201715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1776.340874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1776.399651] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 1777.217670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.35@tcp [ 1777.384625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1778.472499] LNet: Removed LNI 192.168.202.35@tcp [ 1778.515774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1778.617322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1778.652422] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 1779.573578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1779.638206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.35@tcp [ 1779.822296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1779.840782] LNet: 84501:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1779.849950] LNet: 84501:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 1779.862490] LNetError: 84501:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1779.867175] LNetError: 84501:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 1780.903850] LNet: Removed LNI 192.168.202.35@tcp [ 1780.970195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1781.056834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1781.085467] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 1781.925571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.35@tcp [ 1782.103599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1783.145832] LNet: Removed LNI 192.168.202.35@tcp [ 1783.205399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1783.257684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1783.283101] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 1783.911918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1783.956448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1783.983607] LNet: Removed LNI 192.168.202.35@tcp [ 1783.998754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1784.072105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1784.095376] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 1785.031491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1785.116928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1785.165965] LNet: Removed LNI 192.168.202.35@tcp [ 1785.190580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1785.240631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 128 [ 1785.306889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1785.340318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1785.381680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 1000000000000000000000000 [ 1785.446160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1785.475398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1785.519329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer foo [ 1785.584498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1785.616462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1785.667487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 8 [ 1785.684689] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 1786.373021] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1786.410648] LNet: Removed LNI 192.168.202.35@tcp [ 1786.426992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1786.492775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1793.788458] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 13:31:00 (1790098260) [ 1794.165774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1794.214158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1794.306889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1794.326814] LNet: Added LNI 192.168.202.35@tcp231 [8/256/0/180] [ 1796.124587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1796.146806] LNet: 85805:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1796.152521] LNet: 85805:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 8 previous similar messages [ 1796.157539] LNetError: 85805:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1796.165149] LNetError: 85805:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 1796.179224] LNet: Removed LNI 192.168.202.35@tcp231 [ 1796.203723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1796.281512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1796.318557] LNet: Added LNI 192.168.202.35@tcp231 [8/256/0/180] [ 1796.331095] LNet: 85813:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1796.757561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1796.792532] LNet: Removed LNI 192.168.202.35@tcp231 [ 1796.809530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1796.831982] LNet: Added LNI 192.168.202.35@tcp231 [8/256/0/180] [ 1798.336634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1798.358235] LNet: Removed LNI 192.168.202.35@tcp231 [ 1798.373991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1798.468203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1798.498246] LNet: Added LNI 192.168.202.35@tcp231 [8/256/0/0] [ 1798.505224] LNet: 85966:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1798.985807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1799.028426] LNet: Removed LNI 192.168.202.35@tcp231 [ 1799.068722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1799.095148] LNet: Added LNI 192.168.202.35@tcp231 [8/256/0/0] [ 1800.593834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1800.644295] LNet: Removed LNI 192.168.202.35@tcp231 [ 1800.673968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1800.819889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1800.880608] LNet: Added LNI 192.168.202.35@tcp231 [8/256/0/60] [ 1800.896537] LNet: 86117:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1801.679454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1801.716291] LNet: Removed LNI 192.168.202.35@tcp231 [ 1801.749861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1801.778619] LNet: Added LNI 192.168.202.35@tcp231 [8/256/0/60] [ 1803.185339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1803.220942] LNet: Removed LNI 192.168.202.35@tcp231 [ 1803.243533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1803.306794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1803.323220] LNet: Added LNI 192.168.202.35@tcp231 [8/256/0/180] [ 1803.326851] LNet: 86268:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1803.723137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1803.763574] LNet: Removed LNI 192.168.202.35@tcp231 [ 1803.800434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1803.841913] LNet: Added LNI 192.168.202.35@tcp231 [8/256/0/180] [ 1813.241630] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 13:31:20 (1790098280) [ 1813.345919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1813.421977] LNet: Removed LNI 192.168.202.35@tcp231 [ 1813.460028] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1813.564946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1813.587782] LNet: Added LNI 192.168.202.35@tcp232 [8/256/0/180] [ 1814.364657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1814.409951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1814.451056] LNet: Removed LNI 192.168.202.35@tcp232 [ 1814.482233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1814.503190] LNet: Added LNI 192.168.202.35@tcp232 [8/256/0/180] [ 1815.404358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1815.445535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1815.473906] LNet: Removed LNI 192.168.202.35@tcp232 [ 1815.499836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1815.523450] LNet: Added LNI 192.168.202.35@tcp232 [8/256/0/180] [ 1816.445994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1816.488915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1816.549743] LNet: Removed LNI 192.168.202.35@tcp232 [ 1816.565359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1816.617309] LNet: Added LNI 192.168.202.35@tcp232 [8/256/0/180] [ 1817.439280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1817.485435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1817.519647] LNet: Removed LNI 192.168.202.35@tcp232 [ 1817.562625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1824.957514] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 13:31:32 (1790098292) [ 1826.200105] Key type .llcrypt unregistered [ 1826.201692] Key type ._llcrypt unregistered [ 1834.023712] Key type ._llcrypt registered [ 1834.029970] Key type .llcrypt registered [ 1834.371711] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1834.387733] alg: No test for adler32 (adler32-zlib) [ 1835.354138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1835.401757] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 1843.136361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.135@tcp [ 1852.289359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1852.342703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1852.395603] LNet: Added LNI 192.168.202.35@tcp1 [8/256/0/180] [ 1854.002927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp -d 192.168.202.135@tcp -m GET -e local_interrupt -r 2 [ 1854.032573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp -d 192.168.202.135@tcp1 -m GET -e local_interrupt -r 2 [ 1854.058787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp -d 192.168.202.35@tcp -m GET -e local_interrupt -r 2 [ 1854.082946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp -d 192.168.202.35@tcp1 -m GET -e local_interrupt -r 2 [ 1854.119395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp1 -d 192.168.202.135@tcp -m GET -e local_interrupt -r 2 [ 1854.168728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp1 -d 192.168.202.135@tcp1 -m GET -e local_interrupt -r 2 [ 1854.223711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp1 -d 192.168.202.35@tcp -m GET -e local_interrupt -r 2 [ 1854.268811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp1 -d 192.168.202.35@tcp1 -m GET -e local_interrupt -r 2 [ 1854.341786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.35@tcp 192.168.202.135@tcp [ 1854.362210] LNet: There was an unexpected network error while writing to 192.168.202.135: rc = -22 [ 1854.364916] LNet: 1 local NIs in recovery (showing 1): 192.168.202.35@tcp [ 1854.599843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1854.636205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1854.658764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1854.683501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1854.711711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1854.740930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1854.765692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1854.792337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1854.815592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1854.829492] LNet: 88771:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1854.845586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1854.874973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1854.907590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1854.932775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1854.978619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1855.032839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1855.547715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp -d 192.168.202.135@tcp -m GET -e local_dropped -r 2 [ 1855.583324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp -d 192.168.202.135@tcp1 -m GET -e local_dropped -r 2 [ 1855.618794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp -d 192.168.202.35@tcp -m GET -e local_dropped -r 2 [ 1855.645635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp -d 192.168.202.35@tcp1 -m GET -e local_dropped -r 2 [ 1855.670744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp1 -d 192.168.202.135@tcp -m GET -e local_dropped -r 2 [ 1855.700865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp1 -d 192.168.202.135@tcp1 -m GET -e local_dropped -r 2 [ 1855.724976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp1 -d 192.168.202.35@tcp -m GET -e local_dropped -r 2 [ 1855.759074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp1 -d 192.168.202.35@tcp1 -m GET -e local_dropped -r 2 [ 1855.786221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.35@tcp 192.168.202.135@tcp [ 1855.801919] LNet: There was an unexpected network error while writing to 192.168.202.135: rc = -22 [ 1856.026917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1856.060660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1856.090798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1856.121726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1856.148588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1856.177930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1856.207393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1856.235747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1856.261762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1856.277460] LNet: 88893:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1856.298870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1856.338487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1856.363423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1856.390354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1856.414741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1856.438918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1856.901801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp -d 192.168.202.135@tcp -m GET -e local_aborted -r 2 [ 1856.926508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp -d 192.168.202.135@tcp1 -m GET -e local_aborted -r 2 [ 1856.974777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp -d 192.168.202.35@tcp -m GET -e local_aborted -r 2 [ 1857.010555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp -d 192.168.202.35@tcp1 -m GET -e local_aborted -r 2 [ 1857.046388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp1 -d 192.168.202.135@tcp -m GET -e local_aborted -r 2 [ 1857.081881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp1 -d 192.168.202.135@tcp1 -m GET -e local_aborted -r 2 [ 1857.120415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp1 -d 192.168.202.35@tcp -m GET -e local_aborted -r 2 [ 1857.157682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp1 -d 192.168.202.35@tcp1 -m GET -e local_aborted -r 2 [ 1857.202525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.35@tcp 192.168.202.135@tcp [ 1857.220988] LNet: There was an unexpected network error while writing to 192.168.202.135: rc = -22 [ 1857.478405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1857.503490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1857.530947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1857.564257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1857.596965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1857.656448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1857.707318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1857.756640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1857.797691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1857.818643] LNet: 89015:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1857.836508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1857.859930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1857.896773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1857.926600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1857.967847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1858.004634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1858.636785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp -d 192.168.202.135@tcp -m GET -e local_no_route -r 2 [ 1858.680702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp -d 192.168.202.135@tcp1 -m GET -e local_no_route -r 2 [ 1858.712897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp -d 192.168.202.35@tcp -m GET -e local_no_route -r 2 [ 1858.755561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp -d 192.168.202.35@tcp1 -m GET -e local_no_route -r 2 [ 1858.805825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp1 -d 192.168.202.135@tcp -m GET -e local_no_route -r 2 [ 1858.871861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp1 -d 192.168.202.135@tcp1 -m GET -e local_no_route -r 2 [ 1858.917974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp1 -d 192.168.202.35@tcp -m GET -e local_no_route -r 2 [ 1858.952962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp1 -d 192.168.202.35@tcp1 -m GET -e local_no_route -r 2 [ 1858.996855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.35@tcp 192.168.202.135@tcp [ 1859.268640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1859.299728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1859.356694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1859.390677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1859.425935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1859.455282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1859.497076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1859.533665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1859.570932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1859.604726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1859.636953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1859.681352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1859.718622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1859.749829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1859.787316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1860.209473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp -d 192.168.202.135@tcp -m GET -e local_timeout -r 2 [ 1860.239247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp -d 192.168.202.135@tcp1 -m GET -e local_timeout -r 2 [ 1860.269473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp -d 192.168.202.35@tcp -m GET -e local_timeout -r 2 [ 1860.307655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp -d 192.168.202.35@tcp1 -m GET -e local_timeout -r 2 [ 1860.352823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp1 -d 192.168.202.135@tcp -m GET -e local_timeout -r 2 [ 1860.390988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp1 -d 192.168.202.135@tcp1 -m GET -e local_timeout -r 2 [ 1860.438496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp1 -d 192.168.202.35@tcp -m GET -e local_timeout -r 2 [ 1860.476954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp1 -d 192.168.202.35@tcp1 -m GET -e local_timeout -r 2 [ 1860.525886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.35@tcp 192.168.202.135@tcp [ 1860.540333] LNet: There was an unexpected network error while writing to 192.168.202.135: rc = -22 [ 1860.546592] LNet: Skipped 1 previous similar message [ 1860.794761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1860.823586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1860.850855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1860.884615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1860.913272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1860.939644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1860.962817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1861.001987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1861.064673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1861.091727] LNet: 89260:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1861.110930] LNet: 89260:0:(api-ni.c:301:recovery_interval_set()) Skipped 1 previous similar message [ 1861.142494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1861.175421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1861.208313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1861.250670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1861.279381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1861.308258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1861.755484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp -d 192.168.202.135@tcp -m GET -e remote_dropped -r 2 [ 1861.777833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp -d 192.168.202.135@tcp1 -m GET -e remote_dropped -r 2 [ 1861.799631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp -d 192.168.202.35@tcp -m GET -e remote_dropped -r 2 [ 1861.827540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp -d 192.168.202.35@tcp1 -m GET -e remote_dropped -r 2 [ 1861.854555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp1 -d 192.168.202.135@tcp -m GET -e remote_dropped -r 2 [ 1861.883700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp1 -d 192.168.202.135@tcp1 -m GET -e remote_dropped -r 2 [ 1861.917678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp1 -d 192.168.202.35@tcp -m GET -e remote_dropped -r 2 [ 1861.960936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp1 -d 192.168.202.35@tcp1 -m GET -e remote_dropped -r 2 [ 1861.988425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.35@tcp 192.168.202.135@tcp [ 1862.001328] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.135@tcp [ 1862.193953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1862.260785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1862.309696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1862.380376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1862.425255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1862.462862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1862.489405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1862.542687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1862.608522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1862.663782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1862.705257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1862.751974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1862.788752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1862.828891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1862.889148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1869.173396] LNet: 89456:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1869.181289] LNetError: 89456:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1869.188781] LNet: Removed LNI 192.168.202.35@tcp [ 1869.199250] LNet: Removed LNI 192.168.202.35@tcp1 [ 1869.636755] Key type .llcrypt unregistered [ 1869.642291] Key type ._llcrypt unregistered [ 1885.837566] Key type ._llcrypt registered [ 1885.839381] Key type .llcrypt registered [ 1886.898774] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 13:32:34 (1790098354) [ 1887.305640] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1887.325801] alg: No test for adler32 (adler32-zlib) [ 1888.225501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1888.268500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1888.294683] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 1889.106827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1889.123754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.202.35@tcp [ 1889.134972] LNetError: 90434:0:(api-ni.c:4478:lnet_ni_set_healthv()) manually adding local NI 192.168.202.35@tcp to recovery [ 1889.221908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.202.35@tcp [ 1889.355918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1895.596980] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 13:32:42 (1790098362) [ 1895.696499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1895.724363] LNet: 91016:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1895.730412] LNetError: 91016:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1895.754102] LNet: Removed LNI 192.168.202.35@tcp [ 1895.778338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1895.852047] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1895.889926] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 1896.536618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1896.552830] LNet: Added LNI 192.168.202.35@tcp1 [8/256/0/180] [ 1897.850216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.202.35@tcp [ 1897.898239] LNetError: 91289:0:(api-ni.c:4478:lnet_ni_set_healthv()) manually adding local NI 192.168.202.35@tcp to recovery [ 1898.911507] LNet: 1 local NIs in recovery (showing 1): 192.168.202.35@tcp [ 1898.916491] LNet: There was an unexpected network error while writing to 192.168.202.35: rc = -22 [ 1899.213678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.202.35@tcp [ 1907.675963] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 13:32:54 (1790098374) [ 1909.429702] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg235-server [ 1910.985466] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 13:32:58 (1790098378) [ 1912.546345] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg235-server [ 1913.953873] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 13:33:01 (1790098381) [ 1915.570944] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg235-server [ 1916.966826] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 13:33:04 (1790098384) [ 1917.085675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1917.119494] LNet: 92915:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1917.129179] LNetError: 92915:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1917.135084] LNet: Removed LNI 192.168.202.35@tcp [ 1917.154802] LNet: Removed LNI 192.168.202.35@tcp1 [ 1917.186813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1917.260382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1917.299891] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 1917.941493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1918.035788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1918.072829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 1918.203511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1918.236440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 1918.311275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1923.792588] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 13:33:11 (1790098391) [ 1923.928154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1923.973291] LNet: 93652:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1923.985280] LNetError: 93652:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1923.993517] LNetError: 93652:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1924.004347] LNet: Removed LNI 192.168.202.35@tcp [ 1924.021506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1924.057854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1924.077550] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 1929.674757] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 13:33:17 (1790098397) [ 1929.823244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1929.854510] LNet: 94354:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1929.865172] LNetError: 94354:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1929.886706] LNet: Removed LNI 192.168.202.35@tcp [ 1929.903918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1929.942361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1929.960588] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 1935.267721] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 13:33:22 (1790098402) [ 1936.662208] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 1938.292847] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 13:33:25 (1790098405) [ 1938.803339] LNet: 95389:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1938.808921] LNetError: 95389:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1938.825471] LNet: Removed LNI 192.168.202.35@tcp [ 1939.387160] Key type .llcrypt unregistered [ 1939.388704] Key type ._llcrypt unregistered [ 1946.047287] Key type ._llcrypt registered [ 1946.053917] Key type .llcrypt registered [ 1946.269519] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1946.278475] alg: No test for adler32 (adler32-zlib) [ 1947.187796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1947.224127] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 1954.714702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.135@tcp [ 1959.300359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.202.135@tcp [ 1959.571341] LNet: 95805:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1959.579110] LNetError: 95805:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1959.591902] LNet: Removed LNI 192.168.202.35@tcp [ 1960.056541] Key type .llcrypt unregistered [ 1960.060375] Key type ._llcrypt unregistered [ 1970.966400] Key type ._llcrypt registered [ 1970.968380] Key type .llcrypt registered [ 1971.919243] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 1973.484710] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 1974.832709] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 13:34:02 (1790098442) [ 1976.191395] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 [ 1977.592602] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 13:34:05 (1790098445) [ 1979.317370] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg235-server [ 1980.699228] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 13:34:08 (1790098448) [ 1982.367332] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg235-server [ 1983.934954] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 13:34:11 (1790098451) [ 1985.147632] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1985.177233] alg: No test for adler32 (adler32-zlib) [ 1986.264272] Lustre: Lustre: Build Version: 2.17.58_63_gba3a43f [ 1986.474872] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 1988.159301] Key type lgssc registered [ 1989.236678] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2007.784733] Key type lgssc unregistered [ 2008.088460] LNet: 98984:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2008.095758] LNetError: 98984:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2008.110889] LNet: Removed LNI 192.168.202.35@tcp [ 2008.861344] Key type .llcrypt unregistered [ 2008.864378] Key type ._llcrypt unregistered [ 2022.105718] Key type ._llcrypt registered [ 2022.109475] Key type .llcrypt registered [ 2023.298454] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 13:34:50 (1790098490) [ 2023.684447] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2023.700766] alg: No test for adler32 (adler32-zlib) [ 2024.583530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2024.642876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2024.677791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2024.708883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2024.734849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2030.248302] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 13:34:57 (1790098497) [ 2030.548878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2030.600265] LNet: 100406:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2030.633725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2030.675053] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2030.702066] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 2031.687923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2031.701990] LNet: 100550:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2031.705740] LNetError: 100550:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2031.717680] LNet: Removed LNI 192.168.202.35@tcp [ 2031.764308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2031.838447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 2031.858980] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 2032.369407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2032.388208] LNetError: 100578:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2032.399682] LNet: Removed LNI 192.168.202.35@tcp [ 2032.417822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2032.457982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 2032.497584] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 2039.006929] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 13:35:06 (1790098506) [ 2039.110461] LNet: 101160:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2039.121268] LNet: 101160:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 2039.132960] LNetError: 101160:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2039.155942] LNet: Removed LNI 192.168.202.35@tcp [ 2039.772631] Key type .llcrypt unregistered [ 2039.774651] Key type ._llcrypt unregistered [ 2046.082785] Key type ._llcrypt registered [ 2046.091574] Key type .llcrypt registered [ 2046.461102] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2046.544844] alg: No test for adler32 (adler32-zlib) [ 2047.502248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2047.572104] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 2047.575852] LNet: *** cfs_fail_loc=e003, val=100*** [ 2048.119824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.35@tcp [ 2048.142178] LNet: *** cfs_fail_loc=e003, val=100*** [ 2048.149899] LNet: Skipped 2 previous similar messages [ 2048.161244] Ping reply received with mapping: [ 2048.162791] found metadata reply NI[192.168.202.35@tcp] [ 2048.171769] 23 ca a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 2048.175927] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 2048.180262] 5a 5a 5a 5a 5a 5a 5a 5a [ 2048.398883] LNet: 101564:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2048.411073] LNetError: 101564:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2048.439840] LNet: Removed LNI 192.168.202.35@tcp [ 2049.037653] Key type .llcrypt unregistered [ 2049.040591] Key type ._llcrypt unregistered [ 2060.161579] Key type ._llcrypt registered [ 2060.164077] Key type .llcrypt registered [ 2061.303960] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 13:35:28 (1790098528) [ 2061.795316] Key type .llcrypt unregistered [ 2061.797194] Key type ._llcrypt unregistered [ 2067.317239] Key type ._llcrypt registered [ 2067.320635] Key type .llcrypt registered [ 2067.570633] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2067.616750] alg: No test for adler32 (adler32-zlib) [ 2068.810584] LNetError: 102605:0:(api-ni.c:2855:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 2069.210681] Key type .llcrypt unregistered [ 2069.213973] Key type ._llcrypt unregistered [ 2073.956824] Key type ._llcrypt registered [ 2073.958420] Key type .llcrypt registered [ 2075.011856] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 13:35:42 (1790098542) [ 2075.582355] Key type .llcrypt unregistered [ 2075.584421] Key type ._llcrypt unregistered [ 2081.210893] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2081.212884] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2083.772943] Key type ._llcrypt registered [ 2083.774777] Key type .llcrypt registered [ 2083.972323] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2083.986196] alg: No test for adler32 (adler32-zlib) [ 2084.895707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2084.943669] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2093.830849] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 13:36:01 (1790098561) [ 2093.932869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2093.951250] LNet: 104377:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2093.959802] LNetError: 104377:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2093.986944] LNet: Removed LNI 10.1.2.3@tcp [ 2094.024853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2094.812613] Key type .llcrypt unregistered [ 2094.815358] Key type ._llcrypt unregistered [ 2101.250756] Key type ._llcrypt registered [ 2101.252806] Key type .llcrypt registered [ 2101.404048] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2101.414972] alg: No test for adler32 (adler32-zlib) [ 2102.353503] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 2102.414687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.35@tcp [ 2102.556973] LNet: 104670:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2102.563192] LNetError: 104670:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2103.590419] LNet: Removed LNI 192.168.202.35@tcp [ 2104.215442] Key type .llcrypt unregistered [ 2104.220985] Key type ._llcrypt unregistered [ 2115.682795] Key type ._llcrypt registered [ 2115.684950] Key type .llcrypt registered [ 2117.041651] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 13:36:24 (1790098584) [ 2117.643345] Key type .llcrypt unregistered [ 2117.650489] Key type ._llcrypt unregistered [ 2123.855243] Key type ._llcrypt registered [ 2123.860492] Key type .llcrypt registered [ 2124.031159] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2124.050794] alg: No test for adler32 (adler32-zlib) [ 2130.872502] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 13:36:38 (1790098598) [ 2131.078413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2131.117413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2131.207386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2131.224100] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 2132.068374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2132.090512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 2132.117475] LNet: Removed LNI 192.168.202.35@tcp [ 2132.393532] LNet: 106428:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2132.772741] Key type .llcrypt unregistered [ 2132.775788] Key type ._llcrypt unregistered [ 2143.595647] Key type ._llcrypt registered [ 2143.597150] Key type .llcrypt registered [ 2145.282275] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 13:36:52 (1790098612) [ 2145.622410] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2145.637469] alg: No test for adler32 (adler32-zlib) [ 2146.632971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2146.690095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2146.721083] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 2147.345131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.35@tcp [ 2147.414324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.202.35@tcp [ 2147.438497] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2147.443925] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 2147.449112] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2147, 0, 2197 [ 2147.454027] LNet: Conn 0 sched [nconns, cpt]: 2, 0 [ 2147.457676] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2147.461834] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 2147.466368] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2147, 0, 2197 [ 2147.472087] LNet: Conn 1 sched [nconns, cpt]: 2, 0 [ 2152.906536] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 13:37:00 (1790098620) [ 2153.288892] LNet: 107966:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2153.292943] LNetError: 107966:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2153.309613] LNet: Removed LNI 192.168.202.35@tcp [ 2153.774709] Key type .llcrypt unregistered [ 2153.777615] Key type ._llcrypt unregistered [ 2159.559959] Key type ._llcrypt registered [ 2159.565523] Key type .llcrypt registered [ 2159.769805] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2159.789471] alg: No test for adler32 (adler32-zlib) [ 2160.715450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2160.744632] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 2167.929390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.135@tcp [ 2176.468820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2176.535626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2176.557400] LNet: Added LNI 192.168.202.35@tcp1 [8/256/0/180] [ 2177.214165] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2177.410434] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2177.413714] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2177.421372] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2179.770638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 2179.794311] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 2190.044754] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 13:37:37 (1790098657) [ 2190.256201] LNet: 109478:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2190.265614] LNetError: 109478:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2190.270811] LNet: Removed LNI 192.168.202.35@tcp [ 2190.277386] LNet: Removed LNI 192.168.202.35@tcp1 [ 2190.286482] LNet: Removed LNI 10.1.2.3@tcp99 [ 2190.860764] Key type .llcrypt unregistered [ 2190.862420] Key type ._llcrypt unregistered [ 2197.301297] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2197.315268] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2199.970877] Key type ._llcrypt registered [ 2199.974123] Key type .llcrypt registered [ 2200.157795] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2200.172260] alg: No test for adler32 (adler32-zlib) [ 2201.140172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2201.208331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2201.259209] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 2202.159879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2202.179648] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2203.986824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 2204.027183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2204.064368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 2204.085754] LNetError: 110222:0:(peer.c:2366:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 2204.133023] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2204.204440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.35@tcp [ 2204.718494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2204.822871] LNet: 110242:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2204.832524] LNetError: 110242:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2204.836887] LNet: Removed LNI 10.1.2.3@tcp [ 2205.873122] LNet: Removed LNI 192.168.202.35@tcp [ 2206.471782] Key type .llcrypt unregistered [ 2206.476220] Key type ._llcrypt unregistered [ 2217.347182] Key type ._llcrypt registered [ 2217.349528] Key type .llcrypt registered [ 2218.453669] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 13:38:05 (1790098685) [ 2218.929192] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2218.943633] alg: No test for adler32 (adler32-zlib) [ 2219.848314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2219.917617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2219.945268] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 2220.565212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.35@tcp [ 2220.655857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg235-client@tcp [ 2226.134869] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 13:38:13 (1790098693) [ 2226.209762] LNet: 111766:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2226.222437] LNetError: 111766:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2226.265755] LNet: Removed LNI 192.168.202.35@tcp [ 2226.786493] Key type .llcrypt unregistered [ 2226.789131] Key type ._llcrypt unregistered [ 2232.902859] Key type ._llcrypt registered [ 2232.905400] Key type .llcrypt registered [ 2233.141885] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2233.181384] alg: No test for adler32 (adler32-zlib) [ 2234.120547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 2234.143791] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 2234.182251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 2240.128161] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 13:38:27 (1790098707) [ 2241.662846] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 2242.950953] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 13:38:30 (1790098710) [ 2244.109715] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 2245.714892] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 13:38:33 (1790098713) [ 2245.845327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2245.876392] LNet: 113196:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2245.883518] LNetError: 113196:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2245.910930] LNet: Removed LNI 192.168.202.35@tcp [ 2245.929974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2245.999500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2246.018794] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 2246.546844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2246.600817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2251.751618] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 13:38:39 (1790098719) [ 2251.908845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2251.938527] LNet: 113904:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2251.956064] LNetError: 113904:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2253.032477] LNet: Removed LNI 192.168.202.35@tcp [ 2253.074388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2253.134270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 2253.181888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2253.562986] LNet: 113943:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2253.953815] Key type .llcrypt unregistered [ 2253.955359] Key type ._llcrypt unregistered [ 2264.408758] Key type ._llcrypt registered [ 2264.410734] Key type .llcrypt registered [ 2265.623224] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 13:38:53 (1790098733) [ 2265.976519] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2266.003902] alg: No test for adler32 (adler32-zlib) [ 2266.942710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2266.981383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2267.023282] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 2267.626885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2267.678507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.35@tcp [ 2273.589461] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 13:39:00 (1790098740) [ 2273.738521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2273.752841] LNet: 115460:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2273.757835] LNetError: 115460:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2273.769617] LNet: Removed LNI 192.168.202.35@tcp [ 2273.785622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2273.834691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 2273.856260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 2278.755631] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 13:39:06 (1790098746) [ 2278.934993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2278.982954] LNet: 116041:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2278.998678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2279.059819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2279.425910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2279.447976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2279.509727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 2279.769586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2279.793567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2279.845437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2279.874651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2279.917622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2280.205652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2280.240948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2280.296900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 2280.708394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2280.725436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2286.046275] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 13:39:13 (1790098753) [ 2286.138151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2286.173438] LNet: 116672:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2286.181297] LNet: 116672:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 2286.211451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2286.245837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2286.261632] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 2286.820660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2286.839559] LNet: Added LNI 192.168.202.35@tcp2 [8/256/0/180] [ 2287.497493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2287.548587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.35@tcp [ 2287.760602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2287.798936] LNetError: 116954:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2287.814844] LNet: Removed LNI 192.168.202.35@tcp [ 2287.825347] LNet: Removed LNI 192.168.202.35@tcp2 [ 2287.864920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2287.955699] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 2287.960931] LNet: Added LNI 192.168.202.35@tcp2 [8/256/0/180] [ 2287.964465] LNet: 116960:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2288.606353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.35@tcp [ 2288.849752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2288.869407] LNet: 116997:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2288.873526] LNet: 116997:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 2288.878638] LNetError: 116997:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2288.883120] LNetError: 116997:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2288.890173] LNet: Removed LNI 192.168.202.35@tcp [ 2288.895942] LNet: Removed LNI 192.168.202.35@tcp2 [ 2288.926266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2288.973687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2289.001661] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 2289.684278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2289.709780] LNet: Added LNI 192.168.202.35@tcp2 [8/256/0/180] [ 2290.560947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.35@tcp [ 2290.630500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2290.680204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.35@tcp [ 2290.951918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2290.965876] LNetError: 117282:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2290.969787] LNetError: 117282:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2290.983878] LNet: Removed LNI 192.168.202.35@tcp [ 2292.007450] LNet: Removed LNI 192.168.202.35@tcp2 [ 2292.058831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2292.140926] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 2292.144253] LNet: Added LNI 192.168.202.35@tcp2 [8/256/0/180] [ 2292.149953] LNet: 117288:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2292.783830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.35@tcp [ 2292.995477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2293.019504] LNet: 117325:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2293.027447] LNet: 117325:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 2293.036405] LNet: Removed LNI 192.168.202.35@tcp [ 2294.054874] LNet: Removed LNI 192.168.202.35@tcp2 [ 2294.108127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2294.151808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2294.167352] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 2294.779403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2294.798040] LNet: Added LNI 192.168.202.35@tcp2 [8/256/0/180] [ 2295.542944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2295.623360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.35@tcp [ 2295.858530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2295.876866] LNetError: 117605:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2295.881619] LNetError: 117605:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 2296.928621] LNet: Removed LNI 192.168.202.35@tcp [ 2296.938774] LNet: Removed LNI 192.168.202.35@tcp2 [ 2296.960815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2297.024084] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 2297.034991] LNet: Added LNI 192.168.202.35@tcp2 [8/256/0/180] [ 2297.044810] LNet: 117611:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2297.603925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.35@tcp [ 2297.760693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2298.783322] LNet: Removed LNI 192.168.202.35@tcp [ 2298.818985] LNet: Removed LNI 192.168.202.35@tcp2 [ 2298.844755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2298.943875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2298.985437] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 2299.933802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2299.953785] LNet: Added LNI 192.168.202.35@tcp2 [8/256/0/180] [ 2300.752609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.35@tcp [ 2300.794674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2300.863487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.35@tcp [ 2301.046584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2301.068379] LNet: 117933:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2301.072811] LNet: 117933:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 2302.111379] LNet: Removed LNI 192.168.202.35@tcp [ 2303.142945] LNet: Removed LNI 192.168.202.35@tcp2 [ 2303.202421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2303.269632] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 2303.273484] LNet: Added LNI 192.168.202.35@tcp2 [8/256/0/180] [ 2303.276527] LNet: 117939:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2303.730625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.35@tcp [ 2303.884544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2303.908952] LNetError: 117976:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2303.914806] LNetError: 117976:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 2303.927896] LNet: Removed LNI 192.168.202.35@tcp [ 2305.008274] LNet: Removed LNI 192.168.202.35@tcp2 [ 2305.043294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2305.112616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2305.147443] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 2305.758283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2305.777174] LNet: Added LNI 192.168.202.35@tcp2 [8/256/0/180] [ 2306.585375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2306.608268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2306.659309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.35@tcp [ 2306.937409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2307.999259] LNet: Removed LNI 192.168.202.35@tcp [ 2308.012053] LNet: Removed LNI 192.168.202.35@tcp2 [ 2308.049194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2308.113223] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 2308.118829] LNet: Added LNI 192.168.202.35@tcp2 [8/256/0/180] [ 2308.123130] LNet: 118279:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2308.578983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.35@tcp [ 2308.912818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2309.983243] LNet: Removed LNI 192.168.202.35@tcp [ 2310.000373] LNet: Removed LNI 192.168.202.35@tcp2 [ 2310.025509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2310.069248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2310.086929] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 2310.738325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2310.761452] LNet: Added LNI 192.168.202.35@tcp2 [8/256/0/180] [ 2311.714667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.35@tcp [ 2311.768728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2311.828351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2311.879849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.35@tcp [ 2312.238288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2313.311164] LNet: Removed LNI 192.168.202.35@tcp [ 2314.344542] LNet: Removed LNI 192.168.202.35@tcp2 [ 2314.404379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2314.458921] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 2314.464420] LNet: Added LNI 192.168.202.35@tcp2 [8/256/0/180] [ 2314.940651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.35@tcp [ 2321.037324] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 2322.398657] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 13:39:50 (1790098790) [ 2322.470901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2322.487499] LNet: 119392:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2322.497334] LNet: 119392:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 2322.505854] LNetError: 119392:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2322.510571] LNetError: 119392:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 2322.514764] LNet: Removed LNI 192.168.202.35@tcp [ 2323.559211] LNet: Removed LNI 192.168.202.35@tcp2 [ 2323.605043] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2323.875512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2323.892722] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 2324.423215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2324.436290] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2325.193426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2325.209557] LNet: Removed LNI 10.1.2.3@tcp [ 2325.223196] LNet: Removed LNI 192.168.202.35@tcp [ 2325.238513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2325.290400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 2325.310047] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 2325.314899] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2325.625993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 2325.646922] LNet: Removed LNI 192.168.202.35@tcp [ 2325.654954] LNet: Removed LNI 10.1.2.3@tcp [ 2331.410573] Lustre: DEBUG MARKER: == sanity-lnet test 475: per-operation latency stats populate and reset ========================================================== 13:39:59 (1790098799) [ 2331.570326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2331.635984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2331.718541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2331.761420] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 2332.483833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2332.515193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2332.554708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.35@tcp [ 2332.600288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.35@tcp [ 2332.633449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.35@tcp [ 2332.661260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.35@tcp [ 2332.687636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.35@tcp [ 2332.723764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.35@tcp [ 2332.751775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.35@tcp [ 2332.796687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.35@tcp [ 2332.838665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.35@tcp [ 2332.881343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.35@tcp [ 2332.995390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2338.355026] Lustre: DEBUG MARKER: == sanity-lnet test 476: lnetctl set latency_stats toggles capture ========================================================== 13:40:05 (1790098805) [ 2338.451743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2339.495662] LNet: Removed LNI 192.168.202.35@tcp [ 2339.543327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2339.586492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2339.606914] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 2340.316755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 0 [ 2340.346225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2340.387273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.35@tcp [ 2340.428519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.35@tcp [ 2340.470462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.35@tcp [ 2340.522972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.35@tcp [ 2340.557760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.35@tcp [ 2340.592150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.35@tcp [ 2340.615514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.35@tcp [ 2340.642436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.35@tcp [ 2340.671897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.35@tcp [ 2340.698988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.35@tcp [ 2340.772897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2340.806807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.35@tcp [ 2340.848557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.35@tcp [ 2340.880332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.35@tcp [ 2340.908445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.35@tcp [ 2340.932725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.35@tcp [ 2340.971571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.35@tcp [ 2341.011967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.35@tcp [ 2341.053341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.35@tcp [ 2341.086396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.35@tcp [ 2341.134297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.35@tcp [ 2346.405045] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 13:40:13 (1790098813) [ 2346.516076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2347.560615] LNet: Removed LNI 192.168.202.35@tcp [ 2347.595778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2347.728068] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2347.927990] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2347.941677] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2347.957089] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2350.211925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2350.231635] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 2350.875598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2350.898778] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2352.388019] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.35@tcp [ 2352.454833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 2352.504353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.202.35@tcp -m PUT -e local_timeout -r 1 [ 2352.551891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 2352.601525] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2352.606297] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2352.836309] LNetError: 121835:0:(lib-move.c:4794:lnet_parse()) 192.168.202.35@tcp, src 192.168.202.35@tcp: Dropping PUT (error -108 looking up sender) [ 2353.889856] LNet: Removed LNI 10.1.2.3@tcp [ 2354.918716] LNet: Removed LNI 192.168.202.35@tcp [ 2355.310666] Key type .llcrypt unregistered [ 2355.311873] Key type ._llcrypt unregistered [ 2365.744699] Key type ._llcrypt registered [ 2365.747258] Key type .llcrypt registered [ 2366.964066] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 13:40:34 (1790098834) [ 2367.304965] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2367.321779] alg: No test for adler32 (adler32-zlib) [ 2368.246516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2368.585560] LNet: 123145:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2369.047632] Key type .llcrypt unregistered [ 2369.050538] Key type ._llcrypt unregistered [ 2374.219935] Key type ._llcrypt registered [ 2374.224462] Key type .llcrypt registered [ 2374.403504] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2374.423156] alg: No test for adler32 (adler32-zlib) [ 2375.315704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2375.329025] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 2381.464127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.135@tcp [ 2381.679073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 2382.091272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.202.135@tcp [ 2382.313330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2382.338519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2382.363709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2382.390641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2382.415244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2382.437067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2382.466574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2382.489824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2382.495520] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.135@tcp [ 2382.529257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2382.550648] LNet: 123595:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2382.572663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2382.600712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2382.622744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2382.646305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2382.680647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2382.721658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2383.288498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2383.514430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2383.544468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2383.565690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2383.590948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2383.606125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2383.626478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2383.647155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2383.666465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2383.690568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2383.700738] LNet: 123715:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2383.710431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2383.725763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2383.739389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2383.773478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2383.826339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2383.866978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2384.480965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.202.135@tcp [ 2384.645965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2384.670226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2384.689918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2384.716847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2384.740736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2384.766736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2384.790579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2384.816484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2384.847726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2384.867467] LNet: 123834:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2384.882844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2384.904707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2384.920713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2384.937587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2384.969839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2385.005746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2385.557542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2385.716393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2385.737567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2385.761424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2385.789319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2385.817537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2385.845535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2385.874670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2385.903447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2385.941929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2385.965447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2385.989345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2386.019387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2386.049863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2386.098752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2386.148961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2386.416596] LNet: 123999:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2386.420791] LNetError: 123999:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2386.434714] LNet: Removed LNI 192.168.202.35@tcp [ 2386.680695] Key type .llcrypt unregistered [ 2386.684486] Key type ._llcrypt unregistered [ 2399.822384] Key type ._llcrypt registered [ 2399.824649] Key type .llcrypt registered [ 2400.732448] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 13:41:08 (1790098868) [ 2401.036554] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2401.049058] alg: No test for adler32 (adler32-zlib) [ 2402.010551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2402.295285] LNet: 124885:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2402.677461] Key type .llcrypt unregistered [ 2402.679374] Key type ._llcrypt unregistered [ 2408.003685] Key type ._llcrypt registered [ 2408.006890] Key type .llcrypt registered [ 2408.193535] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2408.205759] alg: No test for adler32 (adler32-zlib) [ 2409.083414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2409.107649] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 2416.101429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.135@tcp [ 2423.032033] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2423.096362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2423.124425] LNet: Added LNI 192.168.202.35@tcp1 [8/256/0/180] [ 2423.762130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 2424.296042] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.202.135@tcp [ 2424.482028] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.135@tcp [ 2424.503115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2424.532919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2424.563396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2424.609594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2424.631848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2424.658642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2424.694451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2424.727963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2424.767332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2424.785569] LNet: 125538:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2424.800658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2424.833110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2424.877237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2424.918306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2424.957334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2424.980847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2425.661536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2425.882997] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2425.918715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2425.957128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2425.986803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2426.021939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2426.060321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2426.146397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2426.169656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2426.190477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2426.214123] LNet: 125657:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2426.227399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2426.266977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2426.292359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2426.335184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2426.386138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2426.445092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2427.077456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.202.135@tcp [ 2427.340611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2427.373313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2427.405903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2427.437868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2427.468337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2427.504307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2427.532090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2427.566293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2427.601693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2427.624322] LNet: 125776:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2427.641084] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2427.679152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2427.714914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2427.747158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2427.774206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2427.808516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2428.401135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2428.691596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2428.724085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2428.769037] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2428.788413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2428.818494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2428.848991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2428.890373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2428.940654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2428.978617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2429.015670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2429.041379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2429.073866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2429.112422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2429.135780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2429.169590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2433.629159] LNet: 125976:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2433.633678] LNetError: 125976:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2433.641519] LNet: Removed LNI 192.168.202.35@tcp [ 2433.655052] LNet: Removed LNI 192.168.202.35@tcp1 [ 2433.986975] Key type .llcrypt unregistered [ 2433.990096] Key type ._llcrypt unregistered [ 2447.200738] Key type ._llcrypt registered [ 2447.202252] Key type .llcrypt registered [ 2448.219508] Lustre: DEBUG MARKER: == sanity-lnet test 525: keep reachable peer on local net delete ========================================================== 13:41:55 (1790098915) [ 2448.501875] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2448.517714] alg: No test for adler32 (adler32-zlib) [ 2449.374716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2449.415544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2449.435737] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 2450.041720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2450.050787] LNet: Added LNI 192.168.202.35@tcp2 [8/256/0/180] [ 2450.716542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.35@tcp [ 2450.805763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp [ 2451.871228] LNet: Removed LNI 192.168.202.35@tcp [ 2456.620151] Lustre: DEBUG MARKER: == sanity-lnet test 550: DD-off: keep route up when gw reachable via another local NI ========================================================== 13:42:04 (1790098924) [ 2457.993465] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_550 Need at least 2 local interfaces for multi-rail [ 2459.364620] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 13:42:07 (1790098927) [ 2465.694105] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 13:42:13 (1790098933) [ 2471.772460] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 13:42:19 (1790098939) [ 2477.856356] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 13:42:25 (1790098945) [ 2483.649532] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 13:42:31 (1790098951) [ 2489.898072] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 13:42:37 (1790098957) [ 2497.446264] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 13:42:44 (1790098964) [ 2504.980950] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 13:42:52 (1790098972) [ 2510.985833] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 13:42:58 (1790098978) [ 2516.444340] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 13:43:04 (1790098984) [ 2522.543839] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 13:43:10 (1790098990) [ 2528.305595] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 13:43:15 (1790098995) [ 2534.396894] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 13:43:21 (1790099001) [ 2539.485579] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 13:43:27 (1790099007) [ 2545.604085] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 13:43:33 (1790099013) [ 2551.017088] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 13:43:38 (1790099018) [ 2557.342490] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 13:43:44 (1790099024) [ 2563.480658] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 13:43:51 (1790099031) [ 2563.567427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2563.589759] LNet: 137663:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2563.594900] LNetError: 137663:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2563.609940] LNet: Removed LNI 192.168.202.35@tcp2 [ 2563.635333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2563.764205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2563.784293] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 2564.791239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2564.812396] LNet: 137809:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2564.819995] LNetError: 137809:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2564.835139] LNet: Removed LNI 192.168.202.35@tcp [ 2564.865897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2564.927385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2564.960155] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 2570.242061] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 13:43:57 (1790099037) [ 2575.550817] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 13:44:03 (1790099043) [ 2581.305933] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 13:44:09 (1790099049) [ 2586.389288] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 13:44:14 (1790099054) [ 2591.212091] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 13:44:19 (1790099059) [ 2595.940986] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 13:44:23 (1790099063) [ 2600.635469] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 13:44:28 (1790099068) [ 2604.662759] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 13:44:32 (1790099072) [ 2608.591744] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 13:44:36 (1790099076) [ 2612.355940] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 13:44:40 (1790099080) [ 2616.191429] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 13:44:44 (1790099084) [ 2620.286913] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 13:44:48 (1790099088) [ 2624.026492] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 2355 sec ========== 13:44:52 (1790099092) [ 2624.169469] LNet: 145271:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2624.172725] LNetError: 145271:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2624.183629] LNet: Removed LNI 192.168.202.35@tcp [ 2624.535671] Key type .llcrypt unregistered [ 2624.536862] Key type ._llcrypt unregistered [ 2631.050094] Key type ._llcrypt registered [ 2631.052323] Key type .llcrypt registered [ 2631.292213] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2631.301694] alg: No test for adler32 (adler32-zlib) [ 2632.304533] Lustre: Lustre: Build Version: 2.17.58_63_gba3a43f [ 2632.478941] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 2634.119212] Key type lgssc registered [ 2634.879226] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2675.085772] Lustre: Mounted lustre-client - version 2.17.58_63_gba3a43f [ 2678.429334] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2682.112659] Lustre: Unmounted lustre-client [ 2693.767568] Key type lgssc unregistered [ 2693.926424] LNet: 148222:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2693.929615] LNetError: 148222:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2693.943863] LNet: Removed LNI 192.168.202.35@tcp [ 2694.274135] Key type .llcrypt unregistered [ 2694.275860] Key type ._llcrypt unregistered