[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 471452381 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001013] APIC: Switch to symmetric I/O mode setup [ 0.003350] x2apic enabled [ 0.004010] Switched APIC routing to physical x2apic. [ 0.005015] kvm-guest: setup PV IPIs [ 0.007732] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008025] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010009] pid_max: default: 32768 minimum: 301 [ 0.011155] LSM: Security Framework initializing [ 0.012063] Yama: becoming mindful. [ 0.013044] SELinux: Initializing. [ 0.014086] *** VALIDATE selinux *** [ 0.023431] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.028574] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.030149] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.032033] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.034126] *** VALIDATE tmpfs *** [ 0.035509] *** VALIDATE proc *** [ 0.036282] *** VALIDATE cgroup *** [ 0.037014] *** VALIDATE cgroup2 *** [ 0.038293] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.039153] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.040010] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.041035] Spectre V2 : User space: Vulnerable [ 0.042013] Speculative Store Bypass: Vulnerable [ 0.045372] debug: unmapping init [mem 0xffffffff85e59000-0xffffffff85e60fff] [ 0.047944] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.048731] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.049024] ... version: 2 [ 0.050015] ... bit width: 48 [ 0.051018] ... generic registers: 4 [ 0.052019] ... value mask: 0000ffffffffffff [ 0.053014] ... max period: 00007fffffffffff [ 0.054019] ... fixed-purpose events: 3 [ 0.055019] ... event mask: 000000070000000f [ 0.056303] rcu: Hierarchical SRCU implementation. [ 0.058501] smp: Bringing up secondary CPUs ... [ 0.059625] x86: Booting SMP configuration: [ 0.060038] .... node #0, CPUs: #1 #2 #3 [ 0.063472] smp: Brought up 1 node, 4 CPUs [ 0.065015] smpboot: Max logical packages: 1 [ 0.066027] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.158019] node 0 deferred pages initialised in 90ms [ 0.161101] devtmpfs: initialized [ 0.164320] x86/mm: Memory block size: 128MB [ 0.166855] gcov: version magic: 0x41383552 [ 0.170191] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.174127] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.177367] pinctrl core: initialized pinctrl subsystem [ 0.179203] [ 0.179763] ************************************************************* [ 0.181011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.183014] ** ** [ 0.185016] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.188014] ** ** [ 0.190008] ** This means that this kernel is built to expose internal ** [ 0.192011] ** IOMMU data structures, which may compromise security on ** [ 0.193010] ** your system. ** [ 0.195011] ** ** [ 0.197010] ** If you see this message and you are not debugging the ** [ 0.199021] ** kernel, report this immediately to your vendor! ** [ 0.200007] ** ** [ 0.202009] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.204010] ************************************************************* [ 0.206659] NET: Registered protocol family 16 [ 0.208433] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.211059] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.213073] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.217114] cpuidle: using governor menu [ 0.219000] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.222600] PCI: Using configuration type 1 for base access [ 0.224120] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.234091] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.236025] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.240055] cryptd: max_cpu_qlen set to 1000 [ 0.243228] ACPI: Added _OSI(Module Device) [ 0.245019] ACPI: Added _OSI(Processor Device) [ 0.246029] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.248018] ACPI: Added _OSI(Processor Aggregator Device) [ 0.253730] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.258530] ACPI: Interpreter enabled [ 0.260058] ACPI: PM: (supports S0 S3 S4 S5) [ 0.261011] ACPI: Using IOAPIC for interrupt routing [ 0.263102] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.266442] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.276585] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.279053] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.282019] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.285080] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.290300] acpiphp: Slot [2] registered [ 0.292134] acpiphp: Slot [5] registered [ 0.293114] acpiphp: Slot [6] registered [ 0.295119] acpiphp: Slot [3] registered [ 0.296096] acpiphp: Slot [4] registered [ 0.298123] acpiphp: Slot [7] registered [ 0.299107] acpiphp: Slot [8] registered [ 0.301111] acpiphp: Slot [9] registered [ 0.302114] acpiphp: Slot [10] registered [ 0.304117] acpiphp: Slot [11] registered [ 0.306105] acpiphp: Slot [12] registered [ 0.307116] acpiphp: Slot [13] registered [ 0.308086] acpiphp: Slot [14] registered [ 0.309076] acpiphp: Slot [15] registered [ 0.310088] acpiphp: Slot [16] registered [ 0.312126] acpiphp: Slot [17] registered [ 0.314101] acpiphp: Slot [18] registered [ 0.315118] acpiphp: Slot [19] registered [ 0.317106] acpiphp: Slot [20] registered [ 0.318099] acpiphp: Slot [21] registered [ 0.320100] acpiphp: Slot [22] registered [ 0.321104] acpiphp: Slot [23] registered [ 0.322065] acpiphp: Slot [24] registered [ 0.324106] acpiphp: Slot [25] registered [ 0.325119] acpiphp: Slot [26] registered [ 0.327099] acpiphp: Slot [27] registered [ 0.328094] acpiphp: Slot [28] registered [ 0.330115] acpiphp: Slot [29] registered [ 0.331097] acpiphp: Slot [30] registered [ 0.332101] acpiphp: Slot [31] registered [ 0.334062] PCI host bridge to bus 0000:00 [ 0.335017] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.338030] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.340029] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.342024] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.345018] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.348028] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.350247] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.353088] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.357429] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.363013] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.366879] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.369021] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.370019] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.373018] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.376091] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.379806] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.382041] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.384796] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.388862] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.399659] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.404015] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.409501] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.415015] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.421018] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.447016] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.460040] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.473028] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.479023] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.496031] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.506873] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.508408] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.510293] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.511228] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.513178] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.518132] iommu: Default domain type: Passthrough [ 0.519409] SCSI subsystem initialized [ 0.520102] ACPI: bus type USB registered [ 0.521123] usbcore: registered new interface driver usbfs [ 0.523089] usbcore: registered new interface driver hub [ 0.524066] usbcore: registered new device driver usb [ 0.526149] pps_core: LinuxPPS API ver. 1 registered [ 0.527007] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.529081] PTP clock support registered [ 0.531120] EDAC MC: Ver: 3.0.0 [ 0.532208] PCI: Using ACPI for IRQ routing [ 0.534914] NetLabel: Initializing [ 0.536021] NetLabel: domain hash size = 128 [ 0.538019] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.540099] NetLabel: unlabeled traffic allowed by default [ 0.542034] vgaarb: loaded [ 0.543301] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.544009] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.551329] clocksource: Switched to clocksource kvm-clock [ 0.662355] VFS: Disk quotas dquot_6.6.0 [ 0.663919] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.666699] *** VALIDATE ramfs *** [ 0.668082] *** VALIDATE hugetlbfs *** [ 0.669750] pnp: PnP ACPI init [ 0.672168] pnp: PnP ACPI: found 6 devices [ 0.694201] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.697764] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.700078] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.702261] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.703991] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.705821] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.708833] NET: Registered protocol family 2 [ 0.711497] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.716884] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.720567] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.726306] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.729984] TCP: Hash tables configured (established 65536 bind 65536) [ 0.732920] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.735548] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.738540] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.741727] NET: Registered protocol family 1 [ 0.744757] RPC: Registered named UNIX socket transport module. [ 0.747011] RPC: Registered udp transport module. [ 0.748871] RPC: Registered tcp transport module. [ 0.750721] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.753185] NET: Registered protocol family 44 [ 0.754865] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.756974] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.759243] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.761723] PCI: CLS 0 bytes, default 64 [ 0.763522] Unpacking initramfs... [ 2.152594] debug: unmapping init [mem 0xffff94a93cc64000-0xffff94a93ffcffff] [ 2.158038] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.160377] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.163690] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.656660] Initialise system trusted keyrings [ 2.658499] Key type blacklist registered [ 2.660637] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.670314] zbud: loaded [ 2.673532] *** VALIDATE nfs *** [ 2.674903] *** VALIDATE nfs4 *** [ 2.676701] pstore: using deflate compression [ 2.680502] Platform Keyring initialized [ 2.783678] NET: Registered protocol family 38 [ 2.785848] Key type asymmetric registered [ 2.787502] Asymmetric key parser 'x509' registered [ 2.789583] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.793208] io scheduler mq-deadline registered [ 2.795075] io scheduler kyber registered [ 2.796905] io scheduler bfq registered [ 2.798963] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.802342] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.805197] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.807976] ACPI: Power Button [PWRF] [ 2.813995] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.820707] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.830585] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.859538] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.890642] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.895852] Non-volatile memory driver v1.3 [ 2.897154] Linux agpgart interface v0.103 [ 2.927052] virtio_blk virtio1: [vda] 136032 512-byte logical blocks (69.6 MB/66.4 MiB) [ 2.929894] vda: detected capacity change from 0 to 69648384 [ 2.943476] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 2.946464] vdb: detected capacity change from 0 to 1073741824 [ 2.951907] libphy: Fixed MDIO Bus: probed [ 2.960942] usbcore: registered new interface driver usbserial_generic [ 2.963424] usbserial: USB Serial support registered for generic [ 2.965326] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 2.969856] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 2.972193] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 2.974886] mousedev: PS/2 mouse device common for all mice [ 2.978252] rtc_cmos 00:05: RTC can wake from S4 [ 2.980820] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 2.982246] rtc_cmos 00:05: registered as rtc0 [ 2.987051] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 2.988091] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 2.990345] intel_pstate: CPU model not supported [ 2.994155] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 2.997395] hid: raw HID events driver (C) Jiri Kosina [ 2.999679] usbcore: registered new interface driver usbhid [ 3.001842] usbhid: USB HID core driver [ 3.003697] drop_monitor: Initializing network drop monitor service [ 3.005338] Initializing XFRM netlink socket [ 3.007679] NET: Registered protocol family 10 [ 3.011506] Segment Routing with IPv6 [ 3.013173] NET: Registered protocol family 17 [ 3.014946] mpls_gso: MPLS GSO support [ 3.020886] RAS: Correctable Errors collector initialized. [ 3.024200] AVX version of gcm_enc/dec engaged. [ 3.025273] AES CTR mode by8 optimization enabled [ 3.096701] sched_clock: Marking stable (3096678825, 0)->(3995875288, -899196463) [ 3.099931] registered taskstats version 1 [ 3.102745] Loading compiled-in X.509 certificates [ 3.104732] zswap: loaded using pool lzo/zbud [ 3.127714] Key type big_key registered [ 3.140105] Key type encrypted registered [ 3.141351] ima: No TPM chip found, activating TPM-bypass! [ 3.142602] ima: Allocated hash algorithm: sha1 [ 3.143574] ima: No architecture policies found [ 3.144618] evm: Initialising EVM extended attributes: [ 3.146096] evm: security.selinux [ 3.146745] evm: security.ima [ 3.147441] evm: security.capability [ 3.148188] evm: HMAC attrs: 0x1 [ 3.149667] rtc_cmos 00:05: setting system clock to 2026-06-02 07:03:14 UTC (1780383794) [ 3.154141] debug: unmapping init [mem 0xffffffff86e03000-0xffffffff86ffffff] [ 3.156017] debug: unmapping init [mem 0xffffffff85b82000-0xffffffff85e58fff] [ 3.164066] Write protecting the kernel read-only data: 28672k [ 3.166131] debug: unmapping init [mem 0xffffffff84203000-0xffffffff843fffff] [ 3.167684] debug: unmapping init [mem 0xffffffff84b14000-0xffffffff84bfffff] [ 3.192481] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.197712] systemd[1]: Detected virtualization kvm. [ 3.198991] systemd[1]: Detected architecture x86-64. [ 3.200070] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.223937] systemd[1]: No hostname configured. [ 3.225728] systemd[1]: Set hostname to . [ 3.227754] random: systemd: uninitialized urandom read (16 bytes read) [ 3.229144] systemd[1]: Initializing machine ID from random generator. [ 3.344593] random: systemd: uninitialized urandom read (16 bytes read) [ 3.347280] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 3.353277] random: systemd: uninitialized urandom read (16 bytes read) [ 3.355938] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 3.361731] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket. [ OK ] Reached target Timers. [ OK ] Listening on udev Control Socket. Starting Setup Virtual Console... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Memstrack Anylazing Service. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Slices. Starting Apply Kernel Variables... Starting Create Volatile Files and Directories... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. Starting Journal Service... [ OK ] Reached target Swap. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 3.941277] device-mapper: uevent: version 1.0.3 [ 3.943379] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 4.612824] virtio_net virtio0 ens2: renamed from eth0 [ 4.637778] scsi host0: ata_piix [ 4.652991] random: fast init done [ 4.678856] scsi host1: ata_piix [ 4.685052] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 4.687749] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 9.052312] dracut-initqueue[587]: RTNETLINK answers: File exists [ 9.677429] random: crng init done [ 9.678879] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.013844] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Timers. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Stopped target Local File Systems. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.161549] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.396423] SELinux: Disabled at runtime. [ 11.455606] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.466170] systemd[1]: Detected virtualization kvm. [ 11.467916] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 11.969152] systemd[1]: initrd-switch-root.service: Succeeded. [ 11.972768] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 11.978144] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 11.984349] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 11.986764] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 11.992462] systemd[1]: Starting Journal Service... Starting Journal Service... [ 11.999385] systemd[1]: Mounting Kernel Debug File System... Mounting Kernel Debug File System... [ OK ] Listening on udev Control Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... Starting Apply Kernel Variables... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Mounting Huge Pages File System... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice system-getty.slice. [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Activating swap /dev/disk/by-label/SWAP... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. Mounting POSIX Message Queue File System... Starting Remount Root and Kernel File Systems... [ OK ] Listening on initctl Compatibility Named Pipe. [ 12.143578] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Started Journal Service. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Huge Pages File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted POSIX Message Queue File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ 12.383675] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /mnt. [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 12.634744] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 12.652414] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 12.763647] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 12.785587] EDAC sbridge: Ver: 1.1.2 [ 13.779660] Key type dns_resolver registered [ 14.075765] NFS: Registering the id_resolver key type [ 14.077485] Key type id_resolver registered [ 14.078912] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Basic System. Starting Login Service... [ OK ] Started D-Bus System Message Bus. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Network Manager... [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Started irqbalance daemon. [ OK ] Reached target sshd-keygen.target. Starting Restore /run/initramfs on shutdown... [ OK ] Started Login Service. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... [ OK ] Started OpenSSH server daemon. Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg242-client login: [ 39.704985] libcfs: loading out-of-tree module taints kernel. [ 39.760923] Key type ._llcrypt registered [ 39.764493] Key type .llcrypt registered [ 40.062381] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 40.070325] alg: No test for adler32 (adler32-zlib) [ 41.136188] Lustre: Lustre: Build Version: 2.17.53_57_g7d1c020 [ 41.487503] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 43.135310] Key type lgssc registered [ 43.898779] Lustre: Echo OBD driver; http://www.lustre.org/ [ 170.418483] Lustre: Mounted lustre-client [ 176.221825] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 194.568278] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing check_logdir /tmp/testlogs/ [ 196.064150] Lustre: lustre-OST0000-osc-ffff94a989393800: disconnect after 23s idle [ 201.221854] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing yml_node [ 206.928821] Lustre: DEBUG MARKER: Client: 2.17.53.57 [ 208.547050] hrtimer: interrupt took 3932740 ns [ 210.389701] Lustre: DEBUG MARKER: MDS: 2.17.53.57 [ 214.043804] Lustre: DEBUG MARKER: OSS: 2.17.53.57 [ 215.999538] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Tue Jun 2 03:06:45 EDT 2026 [ 230.768584] Lustre: Unmounted lustre-client [ 248.926266] Key type lgssc unregistered [ 249.188073] LNet: 8403:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 249.197759] LNetError: 8403:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 250.223669] LNet: Removed LNI 192.168.202.42@tcp [ 250.858162] Key type .llcrypt unregistered [ 250.859598] Key type ._llcrypt unregistered [ 278.301335] Key type ._llcrypt registered [ 278.303956] Key type .llcrypt registered [ 278.508102] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 278.514662] alg: No test for adler32 (adler32-zlib) [ 279.446406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 279.525658] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 279.550633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 284.444538] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 284.676903] LNet: 9287:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 284.684530] LNetError: 9287:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 284.703695] LNet: Removed LNI 192.168.202.42@tcp [ 285.410402] Key type .llcrypt unregistered [ 285.411931] Key type ._llcrypt unregistered [ 292.464085] Key type ._llcrypt registered [ 292.466285] Key type .llcrypt registered [ 294.035841] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 03:08:03 (1780384083) [ 294.561671] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 294.581298] alg: No test for adler32 (adler32-zlib) [ 295.495032] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 295.591523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 295.617604] LNet: 9698:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 295.662758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 295.746852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 295.776584] LNet: 9705:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 302.654152] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 03:08:12 (1780384092) [ 302.831727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 302.844440] LNet: 10281:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 302.863446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 302.929771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 309.194730] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 03:08:18 (1780384098) [ 309.303653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 309.331665] LNet: 10861:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 309.352050] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 309.483426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 317.513292] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 03:08:27 (1780384107) [ 317.635315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 317.672671] LNet: 11447:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 317.715688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 317.830783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 325.139236] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 03:08:34 (1780384114) [ 325.409150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 325.497800] LNet: 12036:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 325.628671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 325.784915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 326.508652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 336.541588] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 03:08:46 (1780384126) [ 336.719769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 336.793472] LNet: 12626:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 336.836364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 336.993213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 337.891380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 338.775339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 347.339613] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 03:08:56 (1780384136) [ 347.602428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 347.650102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 347.764479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 354.987879] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 03:09:04 (1780384144) [ 355.095273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 355.125817] LNet: 13808:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 355.132309] LNet: 13808:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 355.166532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 355.268796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 355.314814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 355.679822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 355.710665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 356.138798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 356.210160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 356.810505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 356.875025] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 357.506793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 357.560904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 358.013313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 358.076498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 358.524743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 358.572597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 359.008573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 359.035975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 359.489777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 359.533735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 359.944572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 359.976936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 360.537085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 360.597565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 361.107154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 361.133781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 361.744745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 361.808067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 368.508630] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 03:09:18 (1780384158) [ 368.766923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 368.833339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 368.886458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 368.918628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 374.898504] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 03:09:24 (1780384164) [ 375.119613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 375.158319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 375.217299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 375.248348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 380.426617] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 03:09:30 (1780384170) [ 380.535512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 380.588839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 380.634476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 380.649459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 385.946131] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 03:09:36 (1780384176) [ 386.049720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 386.100457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 386.180171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 386.198442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 392.654706] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 03:09:42 (1780384182) [ 392.797190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 392.831674] LNet: 16838:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 392.845486] LNet: 16838:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 392.874498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 392.981289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 393.037281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 399.207647] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 03:09:49 (1780384189) [ 399.369340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 399.431522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 399.465052] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 399.485474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 405.887137] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 03:09:55 (1780384195) [ 405.982352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 406.037108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 406.135583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 406.496788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 413.627198] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 03:10:03 (1780384203) [ 413.732392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 413.768847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 413.963307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 414.211563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 421.687925] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 03:10:11 (1780384211) [ 421.822354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 421.883399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 422.022347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 422.215175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 428.807915] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 03:10:18 (1780384218) [ 429.049984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 429.128937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 429.523113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 429.787639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 436.949092] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 03:10:26 (1780384226) [ 437.042740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 437.113965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 437.233852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 437.283488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 444.342189] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 03:10:34 (1780384234) [ 444.692989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 444.736838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 444.864120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 444.924134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 452.278632] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 03:10:42 (1780384242) [ 452.380826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 452.444527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 452.564722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 459.376305] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 03:10:49 (1780384249) [ 459.654981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 459.689849] LNet: 22284:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 459.703209] LNet: 22284:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 8 previous similar messages [ 459.731260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 459.827574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 466.807504] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 03:10:56 (1780384256) [ 466.936151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 467.027686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 467.145914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 467.864397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 475.597248] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 03:11:05 (1780384265) [ 475.722651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 475.760195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 475.859293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 475.903793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 482.141730] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 03:11:12 (1780384272) [ 482.401514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 482.462733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 482.551658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 482.599422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 489.988600] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 03:11:19 (1780384279) [ 490.149759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 490.191396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 490.279947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 490.328487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 496.182908] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 03:11:26 (1780384286) [ 496.399325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 496.425715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 496.485419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 496.506584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 503.138423] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 03:11:33 (1780384293) [ 503.342060] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 503.395821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 503.485932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 503.579662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 503.611183] LNetError: 25817:0:(peer.c:2175:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 503.674443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 503.710289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 509.862499] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 03:11:39 (1780384299) [ 510.094338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 510.135382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 510.245833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.61fTwotCl5 [ 510.365814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.61fTwotCl5 [ 510.424436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.61fTwotCl5 [ 516.272757] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 03:11:46 (1780384306) [ 516.597514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 516.654286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 516.750559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 516.792265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 516.857885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 523.120240] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 03:11:53 (1780384313) [ 523.262757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 523.302752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 523.420862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 526.792078] LNet: Message forwarding will be enabled in 2s [ 529.887608] LNet: Message forwarding enabled [ 530.421963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 530.457504] LNet: Message forwarding will stop in 2s to 6s [ 535.064586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 535.239845] LNet: Message forwarding disabled [ 537.814801] LNet: Message forwarding will be enabled in 2s [ 541.420778] LNet: Message forwarding enabled [ 544.970575] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 03:12:14 (1780384334) [ 545.068155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 546.601666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 546.673987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 552.935189] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 03:12:23 (1780384343) [ 553.129725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 553.171111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 553.232868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 553.265769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 553.293213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 553.318893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 553.348930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 553.392743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 553.422387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 553.452749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 553.485940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 553.546372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 553.583215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 553.623434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 553.657829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 553.688412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 553.726236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 553.756137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 553.786277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 553.824964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 553.854635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 553.871224] LNetError: 28887:0:(router.c:624:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 560.306596] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 03:12:30 (1780384350) [ 560.432995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 560.491848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 560.608953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 568.059438] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 03:12:38 (1780384358) [ 569.954415] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg242-server [ 571.598897] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 03:12:41 (1780384361) [ 571.905713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 571.939604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 571.982506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 572.003747] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 573.532263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.202.[43-47]@tcp [ 573.563819] LNet: 30604:0:(router.c:728:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 573.639494] LNetError: 30393:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.202.43@tcp is being used as a gateway but routing feature is not turned on [ 573.666479] LNetError: 30393:0:(lib-lnet.h:1266:lnet_set_route_aliveness()) route to tcp1 through 192.168.202.43@tcp has gone from up to down [ 574.645465] LNetError: 30393:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.202.44@tcp is being used as a gateway but routing feature is not turned on [ 574.662302] LNetError: 30393:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 575.655399] LNetError: 30393:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.202.45@tcp is being used as a gateway but routing feature is not turned on [ 575.662795] LNetError: 30393:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 4 previous similar messages [ 577.699964] LNetError: 30393:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.202.44@tcp is being used as a gateway but routing feature is not turned on [ 577.720933] LNetError: 30393:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 580.224891] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 03:12:50 (1780384370) [ 581.802367] LNetError: 30393:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.202.46@tcp is being used as a gateway but routing feature is not turned on [ 581.813205] LNetError: 30393:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 19 previous similar messages [ 581.856298] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg242-server [ 583.799113] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 03:12:53 (1780384373) [ 584.125304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 584.176709] LNetError: 31538:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 584.227692] LNet: Removed LNI 192.168.202.42@tcp [ 584.283508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 584.408589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 584.449511] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 584.974342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.202.[43-48/2]@tcp [ 584.994694] LNet: 31655:0:(router.c:728:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 585.001527] LNet: 31655:0:(router.c:728:lnet_add_route()) Skipped 4 previous similar messages [ 585.023786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.202.[43-48/2]@tcp [ 585.028287] LNetError: 31542:0:(lib-lnet.h:1266:lnet_set_route_aliveness()) route to tcp103 through 192.168.202.43@tcp has gone from up to down [ 585.047866] LNetError: 31542:0:(lib-lnet.h:1266:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 591.250190] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 03:13:01 (1780384381) [ 591.397572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 591.428937] LNet: 32243:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 591.436234] LNet: 32243:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 14 previous similar messages [ 591.456197] LNetError: 32243:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 592.489412] LNet: Removed LNI 192.168.202.42@tcp [ 592.570394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 592.731927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 592.836795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 592.872862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 592.971369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 593.050706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 593.087488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 593.136493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 593.226554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 593.262711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 593.358134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 593.475561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 593.553546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 593.619820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 593.730745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 593.805818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 593.876236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 593.959145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 594.008977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 594.082584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 594.211910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 594.292304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 594.390428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 594.511390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 594.565656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 594.637111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 594.761610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 594.822433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 594.951800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 595.051870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 595.121588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 595.213515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 595.283433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 595.339774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 595.434382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 602.652209] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 03:13:12 (1780384392) [ 604.670569] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg242-server [ 606.329937] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 03:13:16 (1780384396) [ 608.508130] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg242-server [ 610.128763] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 03:13:20 (1780384400) [ 610.898887] Key type .llcrypt unregistered [ 610.900575] Key type ._llcrypt unregistered [ 617.510126] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 617.672407] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 617.680917] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 617.688659] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 620.402127] Key type ._llcrypt registered [ 620.404939] Key type .llcrypt registered [ 620.691206] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 620.723283] alg: No test for adler32 (adler32-zlib) [ 621.674782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 621.752684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 621.789386] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 622.335293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 622.350525] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 624.081629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 624.106203] LNet: Removed LNI 10.1.2.3@tcp [ 624.210581] LNet: 34366:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 624.214752] LNetError: 34366:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 624.227231] LNet: Removed LNI 192.168.202.42@tcp [ 624.749347] Key type .llcrypt unregistered [ 624.752360] Key type ._llcrypt unregistered [ 635.415712] Key type ._llcrypt registered [ 635.417940] Key type .llcrypt registered [ 636.670501] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 03:13:46 (1780384426) [ 637.264872] Key type .llcrypt unregistered [ 637.267061] Key type ._llcrypt unregistered [ 642.907289] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 643.048360] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 643.063479] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 643.074649] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 645.664056] Key type ._llcrypt registered [ 645.673421] Key type .llcrypt registered [ 645.876564] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 645.901305] alg: No test for adler32 (adler32-zlib) [ 646.814773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 646.877308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 646.904055] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 647.748842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 647.774503] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 650.027233] LNet: 35860:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 650.035447] LNetError: 35860:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 650.040862] LNet: Removed LNI 192.168.202.42@tcp [ 650.056158] LNet: Removed LNI 10.1.2.3@tcp [ 650.736094] Key type .llcrypt unregistered [ 650.740847] Key type ._llcrypt unregistered [ 661.457838] Key type ._llcrypt registered [ 661.460105] Key type .llcrypt registered [ 662.755472] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 03:14:12 (1780384452) [ 663.608752] Key type .llcrypt unregistered [ 663.613351] Key type ._llcrypt unregistered [ 669.505160] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 669.678696] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 669.695430] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 669.702054] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 672.336558] Key type ._llcrypt registered [ 672.341804] Key type .llcrypt registered [ 672.524853] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 672.540474] alg: No test for adler32 (adler32-zlib) [ 673.569336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 673.637013] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 673.667177] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 675.507310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 675.551971] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 677.144813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 677.163954] LNet: Removed LNI 10.1.2.3@tcp [ 677.174486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 677.198987] LNet: Removed LNI 10.1.2.31@tcp [ 677.208939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 677.224273] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 678.092912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 678.113771] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 678.994472] LNet: 37779:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 679.011302] LNetError: 37779:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 679.018204] LNet: Removed LNI 10.1.2.3@tcp [ 679.039400] LNet: Removed LNI 10.1.2.31@tcp [ 679.561556] Key type .llcrypt unregistered [ 679.564460] Key type ._llcrypt unregistered [ 691.568602] Key type ._llcrypt registered [ 691.571468] Key type .llcrypt registered [ 692.765865] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 03:14:42 (1780384482) [ 693.114636] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 693.130424] alg: No test for adler32 (adler32-zlib) [ 694.056484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 694.097288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 694.112443] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 694.610895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 694.626445] LNet: 38734:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 694.631543] LNetError: 38734:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 694.646870] LNet: Removed LNI 192.168.202.42@tcp [ 694.662787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 694.705112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.202.42@tcp [ 694.720543] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 695.282824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.202.42@tcp [ 695.310839] LNet: Removed LNI 192.168.202.42@tcp [ 695.362851] LNet: 38869:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 695.807486] Key type .llcrypt unregistered [ 695.812351] Key type ._llcrypt unregistered [ 705.881226] Key type ._llcrypt registered [ 705.883524] Key type .llcrypt registered [ 707.383409] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 03:14:57 (1780384497) [ 709.734920] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg242-server [ 711.578754] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 03:15:01 (1780384501) [ 712.108362] Key type .llcrypt unregistered [ 712.111240] Key type ._llcrypt unregistered [ 718.182811] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 718.437175] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 718.447318] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 718.453669] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 720.954521] Key type ._llcrypt registered [ 720.957304] Key type .llcrypt registered [ 721.140244] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 721.164614] alg: No test for adler32 (adler32-zlib) [ 722.088412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 722.155029] LNet: Added LNI 192.168.202.42@tcp1 [8/256/0/180] [ 722.912568] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 725.387630] LNet: 40708:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 725.396152] LNetError: 40708:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 725.401618] LNet: Removed LNI 192.168.202.42@tcp1 [ 725.407606] LNet: Removed LNI 10.1.2.3@tcp2 [ 725.991404] Key type .llcrypt unregistered [ 725.994612] Key type ._llcrypt unregistered [ 738.387829] Key type ._llcrypt registered [ 738.392830] Key type .llcrypt registered [ 739.969291] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 03:15:29 (1780384529) [ 740.373232] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 740.404183] alg: No test for adler32 (adler32-zlib) [ 741.344270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 741.468268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 741.517666] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 742.755183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 742.804686] LNet: 41676:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 742.815403] LNetError: 41676:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 742.835693] LNet: Removed LNI 192.168.202.42@tcp [ 742.857461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 742.929665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 742.969386] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 749.375513] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 03:15:39 (1780384539) [ 749.509524] LNet: 42261:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 749.520522] LNetError: 42261:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 749.546227] LNet: Removed LNI 192.168.202.42@tcp [ 750.209830] Key type .llcrypt unregistered [ 750.212657] Key type ._llcrypt unregistered [ 756.187512] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 756.315881] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 756.321682] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 756.329154] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 758.848307] Key type ._llcrypt registered [ 758.852231] Key type .llcrypt registered [ 759.054465] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 759.075831] alg: No test for adler32 (adler32-zlib) [ 760.015451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 760.091246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 760.147041] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 760.893781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 760.931914] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 763.159197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 763.180808] LNet: 43011:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 763.185705] LNetError: 43011:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 763.190625] LNet: Removed LNI 192.168.202.42@tcp [ 763.200664] LNet: Removed LNI 10.1.2.3@tcp [ 763.243423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 763.312149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 763.379929] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 763.384451] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 763.869378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 763.889757] LNet: 43038:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 763.895542] LNetError: 43038:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 763.900273] LNetError: 43038:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 763.904340] LNet: Removed LNI 192.168.202.42@tcp [ 763.918660] LNet: Removed LNI 10.1.2.3@tcp [ 763.935976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 764.013450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 764.056037] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 764.062981] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 764.407282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 764.423167] LNet: Added LNI 192.168.202.42@tcp2 [8/256/0/180] [ 765.374269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 765.389419] LNet: 43200:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 765.393756] LNetError: 43200:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 765.396353] LNetError: 43200:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 765.400865] LNet: Removed LNI 192.168.202.42@tcp [ 765.402593] LNet: Removed LNI 10.1.2.3@tcp [ 765.409935] LNet: Removed LNI 192.168.202.42@tcp2 [ 765.424395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 765.474668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 765.500739] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 765.506736] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 765.511494] LNet: Added LNI 192.168.202.42@tcp2 [8/256/0/180] [ 772.562436] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 03:16:02 (1780384562) [ 772.713634] LNet: 43792:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 772.719316] LNetError: 43792:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 772.723463] LNetError: 43792:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 772.728803] LNet: Removed LNI 192.168.202.42@tcp [ 772.731698] LNet: Removed LNI 10.1.2.3@tcp [ 772.748209] LNet: Removed LNI 192.168.202.42@tcp2 [ 773.348896] Key type .llcrypt unregistered [ 773.354308] Key type ._llcrypt unregistered [ 779.898538] Key type ._llcrypt registered [ 779.901487] Key type .llcrypt registered [ 780.122984] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 780.160033] alg: No test for adler32 (adler32-zlib) [ 781.160537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 781.207642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 781.224932] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 782.040215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 782.064389] LNet: 44192:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 782.093635] LNetError: 44192:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 782.107912] LNet: Removed LNI 192.168.202.42@tcp [ 782.126124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 782.201235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 782.227927] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 782.783300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 782.815982] LNet: 44222:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 782.825717] LNetError: 44222:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 782.840813] LNet: Removed LNI 192.168.202.42@tcp [ 782.862248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 782.945729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 782.967956] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 783.633552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 783.676240] LNet: Removed LNI 192.168.202.42@tcp [ 783.698909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 783.793412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 783.820161] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 784.627455] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 784.836017] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 784.838217] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 784.856397] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 787.294519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 787.313799] LNet: 44331:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 787.320993] LNet: 44331:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 787.326575] LNetError: 44331:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 787.335116] LNetError: 44331:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 787.348656] LNet: Removed LNI 192.168.202.42@tcp [ 787.366223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 787.465732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 787.490163] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 794.783192] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 03:16:24 (1780384584) [ 794.918420] LNet: 44921:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 794.931574] LNetError: 44921:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 794.954588] LNet: Removed LNI 192.168.202.42@tcp [ 795.619582] Key type .llcrypt unregistered [ 795.622767] Key type ._llcrypt unregistered [ 802.866391] Key type ._llcrypt registered [ 802.868707] Key type .llcrypt registered [ 803.115918] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 803.146298] alg: No test for adler32 (adler32-zlib) [ 804.087307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 804.159636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 804.190432] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 805.038264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 805.060478] LNet: 45325:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 805.077148] LNetError: 45325:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 805.099302] LNet: Removed LNI 192.168.202.42@tcp [ 805.130815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 805.258059] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 805.275951] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 812.155368] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 03:16:41 (1780384601) [ 812.242343] LNet: 45909:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 812.251662] LNetError: 45909:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 812.283929] LNet: Removed LNI 192.168.202.42@tcp [ 813.008715] Key type .llcrypt unregistered [ 813.011178] Key type ._llcrypt unregistered [ 819.154110] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 819.349887] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 819.362761] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 819.384385] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 820.884525] Key type ._llcrypt registered [ 820.886553] Key type .llcrypt registered [ 821.072236] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 821.081642] alg: No test for adler32 (adler32-zlib) [ 821.998845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 822.061840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 822.093902] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 822.973507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 822.998426] LNet: 46353:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 823.003942] LNetError: 46353:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 823.028399] LNet: Removed LNI 192.168.202.42@tcp [ 823.055323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 823.152141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 823.184412] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 830.373433] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 03:17:00 (1780384620) [ 830.497429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 830.538338] LNet: 46946:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 830.552290] LNetError: 46946:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 830.575275] LNet: Removed LNI 192.168.202.42@tcp [ 830.612537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 830.739667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 836.884716] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 03:17:06 (1780384626) [ 837.219698] LNet: 47534:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 837.946865] Key type .llcrypt unregistered [ 837.952087] Key type ._llcrypt unregistered [ 844.507926] Key type ._llcrypt registered [ 844.509762] Key type .llcrypt registered [ 844.699344] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 844.715317] alg: No test for adler32 (adler32-zlib) [ 845.644963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 845.694403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 845.731536] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 846.481395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 846.495991] LNet: 47931:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 846.503396] LNetError: 47931:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 846.516522] LNet: Removed LNI 192.168.202.42@tcp [ 846.534762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 846.594978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 846.624351] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 852.879066] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 03:17:22 (1780384642) [ 853.042737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 853.078806] LNet: 48521:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 853.088643] LNetError: 48521:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 853.134083] LNet: Removed LNI 192.168.202.42@tcp [ 853.175642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 853.230126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 858.740586] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 03:17:28 (1780384648) [ 860.138725] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 861.460168] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 03:17:31 (1780384651) [ 862.932295] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 864.354478] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 03:17:34 (1780384654) [ 866.290246] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 867.645935] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 03:17:37 (1780384657) [ 868.841763] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 870.633817] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 03:17:40 (1780384660) [ 872.200959] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 874.000706] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing ========================================================== 03:17:43 (1780384663) [ 874.209435] LNet: 50575:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 874.805502] Key type .llcrypt unregistered [ 874.808044] Key type ._llcrypt unregistered [ 880.801151] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 880.910315] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 880.921251] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 880.927204] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 883.540636] Key type ._llcrypt registered [ 883.543082] Key type .llcrypt registered [ 883.728737] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 883.747883] alg: No test for adler32 (adler32-zlib) [ 884.676568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 884.743718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 884.764480] LNet: Added LNI 192.168.202.42@tcp [8/512/0/180] [ 884.769539] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 884.772432] LNet: 50891:0:(api-ni.c:2864:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 884.866658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 884.886753] LNet: 50910:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 884.892825] LNetError: 50910:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 884.901284] LNet: Removed LNI 192.168.202.42@tcp [ 884.904527] LNetError: 50910:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 884.927909] LNet: Removed LNI 10.1.2.3@tcp [ 884.962920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 885.049359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 885.071027] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 885.133400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 885.177630] LNet: 50934:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 885.189056] LNetError: 50934:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 885.201301] LNet: Removed LNI 192.168.202.42@tcp [ 885.214585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 885.265912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 885.286793] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 885.292253] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 885.295615] LNet: 50942:0:(api-ni.c:2864:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 885.369541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 885.385322] LNet: 50960:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 885.390948] LNetError: 50960:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 885.395169] LNet: Removed LNI 192.168.202.42@tcp [ 885.397589] LNetError: 50960:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 885.417989] LNet: Removed LNI 10.1.2.3@tcp [ 885.433369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 885.475082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 885.499498] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 885.506431] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 885.510756] LNet: 50969:0:(api-ni.c:2864:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 885.696444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 885.713625] LNet: 50995:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 885.718779] LNetError: 50995:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 885.723777] LNet: Removed LNI 192.168.202.42@tcp [ 885.726950] LNetError: 50995:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 885.741275] LNet: Removed LNI 10.1.2.3@tcp [ 885.759555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 885.820466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 885.834684] LNet: Added LNI 192.168.202.42@tcp [8/256/0/182] [ 885.838311] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 885.930674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 885.953523] LNet: 51023:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 885.959517] LNetError: 51023:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 885.965698] LNet: Removed LNI 192.168.202.42@tcp [ 885.972467] LNetError: 51023:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 885.988416] LNet: Removed LNI 10.1.2.3@tcp [ 886.006978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 886.069731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 886.091916] LNet: Added LNI 192.168.202.42@tcp [8/256/0/182] [ 886.095928] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 886.100519] LNet: 51031:0:(api-ni.c:2864:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 886.176263] LNet: Removed LNI 192.168.202.42@tcp [ 886.188250] LNet: Removed LNI 10.1.2.3@tcp [ 886.889240] Key type .llcrypt unregistered [ 886.894576] Key type ._llcrypt unregistered [ 898.067826] Key type ._llcrypt registered [ 898.069952] Key type .llcrypt registered [ 899.355961] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 03:18:09 (1780384689) [ 899.879809] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 899.916963] alg: No test for adler32 (adler32-zlib) [ 900.852624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 900.915204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 900.940192] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 901.784447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.202.42 [ 901.821475] LNet: 52007:0:(config.c:1627:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 901.845575] LNet: Removed LNI 192.168.202.42@tcp [ 908.707637] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 03:18:18 (1780384698) [ 908.845545] LNet: 52573:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 908.860645] LNetError: 52573:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 908.895629] LNet: Removed LNI 192.168.202.42@tcp [ 909.653354] Key type .llcrypt unregistered [ 909.655465] Key type ._llcrypt unregistered [ 916.715778] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 917.003408] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 919.932740] Key type ._llcrypt registered [ 919.938406] Key type .llcrypt registered [ 920.152309] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 920.171694] alg: No test for adler32 (adler32-zlib) [ 921.691150] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 928.049769] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 03:18:38 (1780384718) [ 928.257889] LNet: 53492:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 928.271922] LNetError: 53492:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 928.294387] LNet: Removed LNI 10.1.2.3@tcp [ 928.731638] Key type .llcrypt unregistered [ 928.733684] Key type ._llcrypt unregistered [ 934.031291] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 934.309478] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 934.313424] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 934.319747] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 937.133347] Key type ._llcrypt registered [ 937.142548] Key type .llcrypt registered [ 937.322638] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 937.344973] alg: No test for adler32 (adler32-zlib) [ 938.259627] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 943.665190] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 03:18:53 (1780384733) [ 945.063395] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 946.497995] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 03:18:56 (1780384736) [ 946.645428] LNet: 54677:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 946.660190] LNetError: 54677:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 946.673796] LNet: Removed LNI 10.1.2.3@tcp [ 947.198673] Key type .llcrypt unregistered [ 947.202153] Key type ._llcrypt unregistered [ 952.916058] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 953.187373] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 953.198814] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 953.205934] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 955.901846] Key type ._llcrypt registered [ 955.903366] Key type .llcrypt registered [ 956.075328] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 956.092690] alg: No test for adler32 (adler32-zlib) [ 956.973934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 957.028288] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 965.097581] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 03:19:15 (1780384755) [ 965.536978] LNet: 55907:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 965.545615] LNetError: 55907:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 965.558280] LNet: Removed LNI 10.1.2.3@tcp [ 966.028680] Key type .llcrypt unregistered [ 966.030347] Key type ._llcrypt unregistered [ 971.568527] Key type ._llcrypt registered [ 971.570515] Key type .llcrypt registered [ 971.731960] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 971.742497] alg: No test for adler32 (adler32-zlib) [ 972.663662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 972.688885] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 979.073393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.142@tcp [ 979.837676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.42@tcp -d 192.168.202.142@tcp -m GET -e local_interrupt -r 1 [ 979.874472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.42@tcp -d 192.168.202.42@tcp -m GET -e local_interrupt -r 1 [ 979.905405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.142@tcp [ 979.922442] LNet: There was an unexpected network error while writing to 192.168.202.142: rc = -22 [ 980.176675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 980.209780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 980.242751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 980.269422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 980.315977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 980.361265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 980.392113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 980.424751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 980.440890] LNet: 56345:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 980.456162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 980.489180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 980.524649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 980.544954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 980.569685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 980.598887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 980.634209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 981.183695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.42@tcp -d 192.168.202.142@tcp -m GET -e local_dropped -r 1 [ 981.214921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.42@tcp -d 192.168.202.42@tcp -m GET -e local_dropped -r 1 [ 981.255688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.142@tcp [ 981.270914] LNet: There was an unexpected network error while writing to 192.168.202.142: rc = -22 [ 981.477607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 981.514545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 981.541912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 981.572282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 981.600305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 981.633820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 981.674231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 981.710432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 981.732946] LNet: 56454:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 981.752962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 981.795868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 981.827724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 981.852475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 981.877677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 981.921661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 981.922025] LNet: 1 local NIs in recovery (showing 1): 192.168.202.42@tcp [ 981.961869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 982.465840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.42@tcp -d 192.168.202.142@tcp -m GET -e local_aborted -r 1 [ 982.499347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.42@tcp -d 192.168.202.42@tcp -m GET -e local_aborted -r 1 [ 982.527176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.142@tcp [ 982.542298] LNet: There was an unexpected network error while writing to 192.168.202.142: rc = -22 [ 982.739183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 982.764290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 982.785546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 982.829300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 982.856956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 982.885370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 982.913948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 982.939797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 982.964833] LNet: 56563:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 982.985218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 983.004953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 983.027469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 983.054969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 983.076509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 983.098475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 983.120860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 983.560026] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.42@tcp -d 192.168.202.142@tcp -m GET -e local_no_route -r 1 [ 983.607448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.42@tcp -d 192.168.202.42@tcp -m GET -e local_no_route -r 1 [ 983.645706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.142@tcp [ 983.834211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 983.872954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 983.908867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 983.939360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 983.992759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 984.028792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 984.049462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 984.071142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 984.097681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 984.187505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 984.249980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 984.298578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 984.344853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 984.398100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 984.443234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 985.041860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.42@tcp -d 192.168.202.142@tcp -m GET -e local_timeout -r 1 [ 985.070576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.42@tcp -d 192.168.202.42@tcp -m GET -e local_timeout -r 1 [ 985.123925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.142@tcp [ 985.152836] LNet: There was an unexpected network error while writing to 192.168.202.142: rc = -22 [ 985.160465] LNet: Skipped 2 previous similar messages [ 985.401248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 985.430059] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 985.462680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 985.491874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 985.537322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 985.567145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 985.589404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 985.617697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 985.639938] LNet: 56781:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 985.644401] LNet: 56781:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 985.672989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 985.713110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 985.739058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 985.765181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 985.796519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 985.838671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 985.901962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 986.376075] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.42@tcp -d 192.168.202.142@tcp -m GET -e local_error -r 1 [ 986.399200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.42@tcp -d 192.168.202.42@tcp -m GET -e local_error -r 1 [ 986.420455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.142@tcp [ 986.639348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 986.663325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 986.696710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 986.751706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 986.795870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 986.824735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 986.866529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 986.911627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 986.934988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 986.962395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 986.986625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 987.009766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 987.045381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 987.065162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 987.095698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 987.321388] LNet: 56932:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 987.325487] LNetError: 56932:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 987.340911] LNet: Removed LNI 192.168.202.42@tcp [ 987.684666] Key type .llcrypt unregistered [ 987.687191] Key type ._llcrypt unregistered [ 1003.722735] Key type ._llcrypt registered [ 1003.725815] Key type .llcrypt registered [ 1004.753338] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 03:19:54 (1780384794) [ 1005.634488] Key type .llcrypt unregistered [ 1005.637446] Key type ._llcrypt unregistered [ 1012.159134] Key type ._llcrypt registered [ 1012.160742] Key type .llcrypt registered [ 1012.503856] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1012.525429] alg: No test for adler32 (adler32-zlib) [ 1013.477049] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1013.501319] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 1020.824977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.142@tcp [ 1030.827413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1030.872675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1030.906452] LNet: Added LNI 192.168.202.42@tcp1 [8/256/0/180] [ 1032.100629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.42@tcp -d 192.168.202.142@tcp -m GET -e local_interrupt -r 1 [ 1032.135948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.42@tcp -d 192.168.202.142@tcp1 -m GET -e local_interrupt -r 1 [ 1032.162725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.42@tcp -d 192.168.202.42@tcp -m GET -e local_interrupt -r 1 [ 1032.191783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.42@tcp -d 192.168.202.42@tcp1 -m GET -e local_interrupt -r 1 [ 1032.218894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.42@tcp1 -d 192.168.202.142@tcp -m GET -e local_interrupt -r 1 [ 1032.242846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.42@tcp1 -d 192.168.202.142@tcp1 -m GET -e local_interrupt -r 1 [ 1032.306944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.42@tcp1 -d 192.168.202.42@tcp -m GET -e local_interrupt -r 1 [ 1032.344527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.42@tcp1 -d 192.168.202.42@tcp1 -m GET -e local_interrupt -r 1 [ 1032.399682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.142@tcp [ 1032.442289] LNet: There was an unexpected network error while writing to 192.168.202.142: rc = -22 [ 1032.452354] LNet: 1 local NIs in recovery (showing 1): 192.168.202.42@tcp [ 1032.683899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1032.708993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1032.755844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1032.802940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1032.828952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1032.884323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1032.926502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1032.969648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1032.982935] LNet: 58398:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1032.996401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1033.024271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1033.046688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1033.068581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1033.096574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1033.119880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1033.164413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1033.577986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.42@tcp -d 192.168.202.142@tcp -m GET -e local_dropped -r 1 [ 1033.603884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.42@tcp -d 192.168.202.142@tcp1 -m GET -e local_dropped -r 1 [ 1033.642906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.42@tcp -d 192.168.202.42@tcp -m GET -e local_dropped -r 1 [ 1033.685648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.42@tcp -d 192.168.202.42@tcp1 -m GET -e local_dropped -r 1 [ 1033.717367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.42@tcp1 -d 192.168.202.142@tcp -m GET -e local_dropped -r 1 [ 1033.745985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.42@tcp1 -d 192.168.202.142@tcp1 -m GET -e local_dropped -r 1 [ 1033.769018] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.42@tcp1 -d 192.168.202.42@tcp -m GET -e local_dropped -r 1 [ 1033.796876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.42@tcp1 -d 192.168.202.42@tcp1 -m GET -e local_dropped -r 1 [ 1033.829297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.142@tcp [ 1033.843406] LNet: There was an unexpected network error while writing to 192.168.202.142: rc = -22 [ 1033.850648] LNet: Skipped 2 previous similar messages [ 1034.152470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1034.187645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1034.238387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1034.276693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1034.311597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1034.331705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1034.355739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1034.380341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1034.397677] LNet: 58519:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1034.414580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1034.459514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1034.495725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1034.535264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1034.570667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1034.600489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1034.626687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1035.112520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.42@tcp -d 192.168.202.142@tcp -m GET -e local_aborted -r 1 [ 1035.140779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.42@tcp -d 192.168.202.142@tcp1 -m GET -e local_aborted -r 1 [ 1035.170768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.42@tcp -d 192.168.202.42@tcp -m GET -e local_aborted -r 1 [ 1035.194096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.42@tcp -d 192.168.202.42@tcp1 -m GET -e local_aborted -r 1 [ 1035.214505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.42@tcp1 -d 192.168.202.142@tcp -m GET -e local_aborted -r 1 [ 1035.236507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.42@tcp1 -d 192.168.202.142@tcp1 -m GET -e local_aborted -r 1 [ 1035.262993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.42@tcp1 -d 192.168.202.42@tcp -m GET -e local_aborted -r 1 [ 1035.306615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.42@tcp1 -d 192.168.202.42@tcp1 -m GET -e local_aborted -r 1 [ 1035.334205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.142@tcp [ 1035.350964] LNet: There was an unexpected network error while writing to 192.168.202.142: rc = -22 [ 1035.355508] LNet: Skipped 2 previous similar messages [ 1035.520805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1035.547282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1035.587659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1035.610362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1035.637453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1035.670975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1035.705561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1035.737483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1035.758352] LNet: 58640:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1035.776215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1035.802273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1035.844657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1035.876910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1035.905994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1035.928556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1035.947969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1036.416566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.42@tcp -d 192.168.202.142@tcp -m GET -e local_no_route -r 1 [ 1036.449466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.42@tcp -d 192.168.202.142@tcp1 -m GET -e local_no_route -r 1 [ 1036.476910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.42@tcp -d 192.168.202.42@tcp -m GET -e local_no_route -r 1 [ 1036.506317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.42@tcp -d 192.168.202.42@tcp1 -m GET -e local_no_route -r 1 [ 1036.537574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.42@tcp1 -d 192.168.202.142@tcp -m GET -e local_no_route -r 1 [ 1036.577442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.42@tcp1 -d 192.168.202.142@tcp1 -m GET -e local_no_route -r 1 [ 1036.632211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.42@tcp1 -d 192.168.202.42@tcp -m GET -e local_no_route -r 1 [ 1036.688600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.42@tcp1 -d 192.168.202.42@tcp1 -m GET -e local_no_route -r 1 [ 1036.751481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.142@tcp [ 1036.967959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1036.997883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1037.028714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1037.045494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1037.062286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1037.090099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1037.125875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1037.164298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1037.195615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1037.224207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1037.253188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1037.284385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1037.315441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1037.343145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1037.370899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1037.815593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.42@tcp -d 192.168.202.142@tcp -m GET -e local_timeout -r 1 [ 1037.841794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.42@tcp -d 192.168.202.142@tcp1 -m GET -e local_timeout -r 1 [ 1037.866625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.42@tcp -d 192.168.202.42@tcp -m GET -e local_timeout -r 1 [ 1037.890654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.42@tcp -d 192.168.202.42@tcp1 -m GET -e local_timeout -r 1 [ 1037.914509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.42@tcp1 -d 192.168.202.142@tcp -m GET -e local_timeout -r 1 [ 1037.946116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.42@tcp1 -d 192.168.202.142@tcp1 -m GET -e local_timeout -r 1 [ 1037.983948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.42@tcp1 -d 192.168.202.42@tcp -m GET -e local_timeout -r 1 [ 1038.022574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.42@tcp1 -d 192.168.202.42@tcp1 -m GET -e local_timeout -r 1 [ 1038.062529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.142@tcp [ 1038.090795] LNet: There was an unexpected network error while writing to 192.168.202.142: rc = -22 [ 1038.097325] LNet: Skipped 5 previous similar messages [ 1038.363707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1038.391707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1038.417985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1038.442336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1038.467192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1038.492744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1038.516969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1038.541509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1038.557561] LNet: 58883:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1038.561264] LNet: 58883:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 1038.577264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1038.603273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1038.625390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1038.670397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1038.703379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1038.738849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1038.761296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1039.295148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.42@tcp -d 192.168.202.142@tcp -m GET -e local_error -r 1 [ 1039.336576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.42@tcp -d 192.168.202.142@tcp1 -m GET -e local_error -r 1 [ 1039.370943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.42@tcp -d 192.168.202.42@tcp -m GET -e local_error -r 1 [ 1039.401934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.42@tcp -d 192.168.202.42@tcp1 -m GET -e local_error -r 1 [ 1039.429528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.42@tcp1 -d 192.168.202.142@tcp -m GET -e local_error -r 1 [ 1039.456493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.42@tcp1 -d 192.168.202.142@tcp1 -m GET -e local_error -r 1 [ 1039.482603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.42@tcp1 -d 192.168.202.42@tcp -m GET -e local_error -r 1 [ 1039.508902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.42@tcp1 -d 192.168.202.42@tcp1 -m GET -e local_error -r 1 [ 1039.533469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.142@tcp [ 1039.704867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1039.733767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1039.762886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1039.785159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1039.814390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1039.842302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1039.870895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1039.902842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1039.936916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1039.970510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1040.007946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1040.042477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1040.080512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1040.122154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1040.157291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1044.507615] LNet: 59080:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1044.512062] LNetError: 59080:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1045.537969] LNet: Removed LNI 192.168.202.42@tcp [ 1045.541138] LNetError: 59080:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1045.554983] LNet: Removed LNI 192.168.202.42@tcp1 [ 1046.005597] Key type .llcrypt unregistered [ 1046.007745] Key type ._llcrypt unregistered [ 1059.049988] Key type ._llcrypt registered [ 1059.051601] Key type .llcrypt registered [ 1060.155590] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 03:20:50 (1780384850) [ 1060.840484] Key type .llcrypt unregistered [ 1060.843225] Key type ._llcrypt unregistered [ 1065.198821] Key type ._llcrypt registered [ 1065.200465] Key type .llcrypt registered [ 1065.383779] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1065.393449] alg: No test for adler32 (adler32-zlib) [ 1066.324278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1066.348104] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 1073.146231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.142@tcp [ 1073.685457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.42@tcp -d 192.168.202.142@tcp -m GET -e remote_dropped -r 1 [ 1073.708414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.42@tcp -d 192.168.202.42@tcp -m GET -e remote_dropped -r 1 [ 1073.735409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.142@tcp [ 1073.747434] LNet: There was an unexpected network error while writing to 192.168.202.142: rc = -22 [ 1073.891090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1073.911876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1073.934644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1073.960967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1073.983612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1074.017632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1074.044229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1074.061469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1074.075766] LNet: 60331:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1074.103975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1074.129076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1074.169168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1074.206446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1074.253323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1074.295261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1074.340351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1074.903320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.42@tcp -d 192.168.202.142@tcp -m GET -e remote_error -r 1 [ 1074.953946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.42@tcp -d 192.168.202.42@tcp -m GET -e remote_error -r 1 [ 1074.996165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.142@tcp [ 1075.010442] LNet: There was an unexpected network error while writing to 192.168.202.142: rc = -22 [ 1075.172548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1075.186845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1075.203803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1075.244270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1075.287643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1075.330217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1075.369946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1075.391487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1075.404871] LNet: 60440:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1075.420560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1075.442797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1075.466851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1075.487297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1075.507523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1075.531850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1075.561343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1075.931647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.42@tcp -d 192.168.202.142@tcp -m GET -e remote_timeout -r 1 [ 1075.966370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.42@tcp -d 192.168.202.42@tcp -m GET -e remote_timeout -r 1 [ 1075.998371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.142@tcp [ 1076.017282] LNet: There was an unexpected network error while writing to 192.168.202.142: rc = -22 [ 1076.222570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1076.255974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1076.282450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1076.327992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1076.366875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1076.428942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1076.467664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1076.501662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1076.514979] LNet: 60549:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1076.533952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1076.579686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1076.601302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1076.630454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1076.668635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1076.684573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1076.704829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1076.932891] LNet: 60590:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1076.938367] LNetError: 60590:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1076.959094] LNet: Removed LNI 192.168.202.42@tcp [ 1077.256653] Key type .llcrypt unregistered [ 1077.259591] Key type ._llcrypt unregistered [ 1092.019054] Key type ._llcrypt registered [ 1092.020600] Key type .llcrypt registered [ 1093.215947] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 03:21:23 (1780384883) [ 1094.025324] Key type .llcrypt unregistered [ 1094.027635] Key type ._llcrypt unregistered [ 1099.313603] Key type ._llcrypt registered [ 1099.315698] Key type .llcrypt registered [ 1099.496458] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1099.505703] alg: No test for adler32 (adler32-zlib) [ 1100.418122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1100.446329] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 1106.597641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.142@tcp [ 1112.283727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1112.317353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1112.339567] LNet: Added LNI 192.168.202.42@tcp1 [8/256/0/180] [ 1113.386691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.42@tcp -d 192.168.202.142@tcp -m GET -e remote_dropped -r 1 [ 1113.409157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.42@tcp -d 192.168.202.142@tcp1 -m GET -e remote_dropped -r 1 [ 1113.430467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.42@tcp -d 192.168.202.42@tcp -m GET -e remote_dropped -r 1 [ 1113.454589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.42@tcp -d 192.168.202.42@tcp1 -m GET -e remote_dropped -r 1 [ 1113.485863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.42@tcp1 -d 192.168.202.142@tcp -m GET -e remote_dropped -r 1 [ 1113.518546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.42@tcp1 -d 192.168.202.142@tcp1 -m GET -e remote_dropped -r 1 [ 1113.551750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.42@tcp1 -d 192.168.202.42@tcp -m GET -e remote_dropped -r 1 [ 1113.588534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.42@tcp1 -d 192.168.202.42@tcp1 -m GET -e remote_dropped -r 1 [ 1113.625132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.142@tcp [ 1113.645608] LNet: There was an unexpected network error while writing to 192.168.202.142: rc = -22 [ 1113.654416] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.142@tcp [ 1113.806636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1113.829051] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1113.854882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1113.882947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1113.914834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1113.939790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1113.967782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1113.990876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1114.006702] LNet: 62056:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1114.022202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1114.049117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1114.075915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1114.098763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1114.124787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1114.151702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1114.177143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1114.202777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1114.676769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.42@tcp -d 192.168.202.142@tcp -m GET -e remote_error -r 1 [ 1114.713604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.42@tcp -d 192.168.202.142@tcp1 -m GET -e remote_error -r 1 [ 1114.738699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.42@tcp -d 192.168.202.42@tcp -m GET -e remote_error -r 1 [ 1114.766232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.42@tcp -d 192.168.202.42@tcp1 -m GET -e remote_error -r 1 [ 1114.797121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.42@tcp1 -d 192.168.202.142@tcp -m GET -e remote_error -r 1 [ 1114.827246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.42@tcp1 -d 192.168.202.142@tcp1 -m GET -e remote_error -r 1 [ 1114.855915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.42@tcp1 -d 192.168.202.42@tcp -m GET -e remote_error -r 1 [ 1114.889941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.42@tcp1 -d 192.168.202.42@tcp1 -m GET -e remote_error -r 1 [ 1114.922796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.142@tcp [ 1114.938753] LNet: There was an unexpected network error while writing to 192.168.202.142: rc = -22 [ 1114.944417] LNet: Skipped 2 previous similar messages [ 1115.147978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1115.171515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1115.198710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1115.220626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1115.240902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1115.261815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1115.284747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1115.306454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1115.318282] LNet: 62180:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1115.332328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1115.357715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1115.383046] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1115.410866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1115.455541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1115.497633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1115.533742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1115.567974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1116.040934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.42@tcp -d 192.168.202.142@tcp -m GET -e remote_timeout -r 1 [ 1116.075411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.42@tcp -d 192.168.202.142@tcp1 -m GET -e remote_timeout -r 1 [ 1116.104850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.42@tcp -d 192.168.202.42@tcp -m GET -e remote_timeout -r 1 [ 1116.137791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.42@tcp -d 192.168.202.42@tcp1 -m GET -e remote_timeout -r 1 [ 1116.178178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.42@tcp1 -d 192.168.202.142@tcp -m GET -e remote_timeout -r 1 [ 1116.209677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.42@tcp1 -d 192.168.202.142@tcp1 -m GET -e remote_timeout -r 1 [ 1116.257594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.42@tcp1 -d 192.168.202.42@tcp -m GET -e remote_timeout -r 1 [ 1116.297429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.42@tcp1 -d 192.168.202.42@tcp1 -m GET -e remote_timeout -r 1 [ 1116.340755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.142@tcp [ 1116.368703] LNet: There was an unexpected network error while writing to 192.168.202.142: rc = -22 [ 1116.529951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1116.595624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1116.633897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1116.654974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1116.682575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1116.703516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1116.735494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1116.762596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1116.777183] LNet: 62303:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1116.793478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1116.826427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1116.854665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1116.889910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1116.931459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1116.964412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1116.992297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1117.021455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1119.936770] LNet: 62381:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1119.942122] LNetError: 62381:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1119.949832] LNet: Removed LNI 192.168.202.42@tcp [ 1119.960922] LNet: Removed LNI 192.168.202.42@tcp1 [ 1120.280591] Key type .llcrypt unregistered [ 1120.282124] Key type ._llcrypt unregistered [ 1132.351194] Key type ._llcrypt registered [ 1132.352464] Key type .llcrypt registered [ 1133.416498] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 03:22:03 (1780384923) [ 1134.844655] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 1136.399716] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 03:22:06 (1780384926) [ 1136.960485] Key type .llcrypt unregistered [ 1136.966232] Key type ._llcrypt unregistered [ 1143.069093] Key type ._llcrypt registered [ 1143.070774] Key type .llcrypt registered [ 1143.268761] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1143.287366] alg: No test for adler32 (adler32-zlib) [ 1144.128545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1144.143958] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 1150.727526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.142@tcp [ 1151.184987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.42@tcp -d 192.168.202.142@tcp -m GET -e network_timeout -r 1 [ 1151.211236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.42@tcp -d 192.168.202.42@tcp -m GET -e network_timeout -r 1 [ 1151.250183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.142@tcp [ 1151.283869] LNet: There was an unexpected network error while writing to 192.168.202.142: rc = -22 [ 1151.332272] LNet: 1 local NIs in recovery (showing 1): 192.168.202.42@tcp [ 1151.477938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1151.506448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1151.529508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1151.553530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1151.574103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1151.590830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1151.606199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1151.627110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1151.643631] LNet: 63923:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1151.664340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1151.679899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1151.695779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1151.723799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1151.757815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1151.784360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1152.024262] LNet: 63962:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1152.031478] LNetError: 63962:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1152.044876] LNet: Removed LNI 192.168.202.42@tcp [ 1152.381433] Key type .llcrypt unregistered [ 1152.383843] Key type ._llcrypt unregistered [ 1165.823430] Key type ._llcrypt registered [ 1165.827783] Key type .llcrypt registered [ 1165.960993] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1165.977259] alg: No test for adler32 (adler32-zlib) [ 1166.899382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1166.914641] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 1173.280502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.142@tcp [ 1180.079975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1180.118447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1180.135530] LNet: Added LNI 192.168.202.42@tcp1 [8/256/0/180] [ 1181.245351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.42@tcp -d 192.168.202.142@tcp -m GET -e network_timeout -r 1 [ 1181.273082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.42@tcp -d 192.168.202.142@tcp1 -m GET -e network_timeout -r 1 [ 1181.306576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.42@tcp -d 192.168.202.42@tcp -m GET -e network_timeout -r 1 [ 1181.333921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.42@tcp -d 192.168.202.42@tcp1 -m GET -e network_timeout -r 1 [ 1181.363322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.42@tcp1 -d 192.168.202.142@tcp -m GET -e network_timeout -r 1 [ 1181.393798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.42@tcp1 -d 192.168.202.142@tcp1 -m GET -e network_timeout -r 1 [ 1181.421516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.42@tcp1 -d 192.168.202.42@tcp -m GET -e network_timeout -r 1 [ 1181.449526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.42@tcp1 -d 192.168.202.42@tcp1 -m GET -e network_timeout -r 1 [ 1181.476929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.142@tcp [ 1181.491913] LNet: There was an unexpected network error while writing to 192.168.202.142: rc = -22 [ 1181.689911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1181.718178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1181.742494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1181.761917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1181.785962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1181.807159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1181.829192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1181.855660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1181.868359] LNet: 64839:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1181.887903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1181.916380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1181.945350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1181.970388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1181.997643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1182.022931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1185.757897] LNet: There was an unexpected network error while writing to 192.168.202.142: rc = -22 [ 1185.980064] LNet: 64913:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1185.983410] LNetError: 64913:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1185.987317] LNet: Removed LNI 192.168.202.42@tcp [ 1186.000156] LNet: Removed LNI 192.168.202.42@tcp1 [ 1186.312816] Key type .llcrypt unregistered [ 1186.317184] Key type ._llcrypt unregistered [ 1199.854599] Key type ._llcrypt registered [ 1199.857507] Key type .llcrypt registered [ 1200.915423] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 03:23:10 (1780384990) [ 1201.220819] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1201.234647] alg: No test for adler32 (adler32-zlib) [ 1202.088333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1202.134713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1202.152625] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 1202.635833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1202.654488] LNet: Added LNI 192.168.202.42@tcp1 [8/256/0/180] [ 1203.477658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.42@tcp [ 1203.560929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1203.616830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1203.659315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1203.685444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1203.712849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1203.743847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.202.42@tcp [ 1203.778292] LNetError: 66012:0:(api-ni.c:4350:lnet_ni_set_healthv()) manually adding local NI 192.168.202.42@tcp to recovery [ 1204.194042] LNet: 1 local NIs in recovery (showing 1): 192.168.202.42@tcp [ 1204.195373] LNet: There was an unexpected network error while writing to 192.168.202.42: rc = -22 [ 1206.243805] LNet: There was an unexpected network error while writing to 192.168.202.42: rc = -22 [ 1207.218359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1210.337610] LNet: There was an unexpected network error while writing to 192.168.202.42: rc = -22 [ 1210.730319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1210.759745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1210.807876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1210.829712] LNet: 66119:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1210.843621] LNetError: 66119:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1211.873280] LNet: Removed LNI 192.168.202.42@tcp [ 1211.875859] LNetError: 66119:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1211.885786] LNet: Removed LNI 192.168.202.42@tcp1 [ 1211.905499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1211.934051] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1211.987720] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 1212.490271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1212.505699] LNet: Added LNI 192.168.202.42@tcp1 [8/256/0/180] [ 1213.100461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.42@tcp [ 1213.126448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1213.175692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1213.204811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1213.231201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1213.257747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1213.291834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1213.317390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.202.42@tcp [ 1213.333668] LNetError: 66391:0:(api-ni.c:4350:lnet_ni_set_healthv()) manually adding local NI 192.168.202.42@tcp to recovery [ 1213.987759] LNet: There was an unexpected network error while writing to 192.168.202.42: rc = -22 [ 1215.670180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1219.106292] LNet: There was an unexpected network error while writing to 192.168.202.42: rc = -22 [ 1219.112502] LNet: Skipped 1 previous similar message [ 1223.479980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1223.506827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1223.530585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1228.455718] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 03:23:38 (1780385018) [ 1228.698237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1228.713712] LNet: 67097:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1228.718548] LNetError: 67097:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1228.726739] LNet: Removed LNI 192.168.202.42@tcp [ 1228.741812] LNet: Removed LNI 192.168.202.42@tcp1 [ 1228.752926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1228.789591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1228.806282] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 1229.222121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1229.239238] LNet: Added LNI 192.168.202.42@tcp1 [8/256/0/180] [ 1229.937323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.42@tcp [ 1229.998225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1230.026230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1230.071908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1230.098673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1230.128794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1230.158418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.202.42@tcp --health 0 [ 1230.262670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1230.816112] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.42@tcp [ 1231.843491] LNet: There was an unexpected network error while writing to 192.168.202.42: rc = -22 [ 1231.857767] LNet: Skipped 1 previous similar message [ 1245.436802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1245.589952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1245.629436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1245.677792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.202.42@tcp --health 500 [ 1245.751552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1251.365751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1251.461849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1251.481742] LNet: 67581:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1251.493078] LNetError: 67581:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1251.496735] LNetError: 67581:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1251.503586] LNet: Removed LNI 192.168.202.42@tcp [ 1251.512325] LNet: Removed LNI 192.168.202.42@tcp1 [ 1251.554298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1251.610643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1251.671895] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 1252.358835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1252.374911] LNet: Added LNI 192.168.202.42@tcp1 [8/256/0/180] [ 1253.114370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.42@tcp [ 1253.140719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1253.206931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1253.256703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1253.292535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1253.328651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1253.363964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1253.394732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.202.42@tcp --health 0 [ 1254.623618] LNet: There was an unexpected network error while writing to 192.168.202.42: rc = -22 [ 1254.630322] LNet: Skipped 3 previous similar messages [ 1255.826388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1265.828953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1265.850542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1265.872781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1271.025232] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 03:24:21 (1780385061) [ 1271.384522] LNet: 68638:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1271.392987] LNetError: 68638:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1271.398419] LNetError: 68638:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1271.403279] LNet: Removed LNI 192.168.202.42@tcp [ 1271.414190] LNet: Removed LNI 192.168.202.42@tcp1 [ 1271.879058] Key type .llcrypt unregistered [ 1271.880168] Key type ._llcrypt unregistered [ 1276.525697] Key type ._llcrypt registered [ 1276.527665] Key type .llcrypt registered [ 1276.646832] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1276.663905] alg: No test for adler32 (adler32-zlib) [ 1277.536873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1277.554933] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 1283.488570] Lustre: DEBUG MARKER: Initial discovery [ 1283.538131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.202.142@tcp [ 1285.424225] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1285.649128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1285.672396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.202.142@tcp [ 1285.692747] LNet: There was an unexpected network error while writing to 192.168.202.142: rc = -22 [ 1285.712856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1285.730845] LNet: 1 local NIs in recovery (showing 1): 192.168.202.42@tcp [ 1285.747492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.202.142@tcp -r 1 -m GET -l 3 [ 1285.777312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.202.142@tcp -r 1 -m GET -e local_error [ 1285.795283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.202.142@tcp -r 1 -m PUT -l 6 [ 1286.965223] Lustre: DEBUG MARKER: Force oleg242-client.virtnet to discover 192.168.202.142@tcp (in background) [ 1287.044535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.202.142@tcp [ 1288.092338] Lustre: DEBUG MARKER: Force oleg242-server to discover 192.168.202.42@tcp [ 1290.088872] LNet: There was an unexpected network error while writing to 192.168.202.142: rc = -22 [ 1290.432514] Lustre: DEBUG MARKER: Wait for 69392 [ 1291.742202] Lustre: DEBUG MARKER: Finished wait on 69392 [ 1291.776143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay del -a [ 1291.815995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1291.823562] LNet: There was an unexpected network error while writing to 192.168.202.142: rc = -22 [ 1292.079734] LNet: 69848:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1292.088044] LNetError: 69848:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1293.164942] LNet: Removed LNI 192.168.202.42@tcp [ 1293.537974] Key type .llcrypt unregistered [ 1293.543651] Key type ._llcrypt unregistered [ 1306.131971] Key type ._llcrypt registered [ 1306.137476] Key type .llcrypt registered [ 1307.286464] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 03:24:57 (1780385097) [ 1307.691647] Key type .llcrypt unregistered [ 1307.692764] Key type ._llcrypt unregistered [ 1313.187643] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1313.374418] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1313.377395] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1313.380122] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1315.918278] Key type ._llcrypt registered [ 1315.920017] Key type .llcrypt registered [ 1316.093791] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1316.110282] alg: No test for adler32 (adler32-zlib) [ 1317.060511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1317.101975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1317.124382] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 1317.688414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1317.700650] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1324.362797] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 03:25:14 (1780385114) [ 1324.498404] LNet: 71926:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1324.507114] LNetError: 71926:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1324.513718] LNet: Removed LNI 192.168.202.42@tcp [ 1324.526312] LNet: Removed LNI 10.1.2.3@tcp [ 1325.052215] Key type .llcrypt unregistered [ 1325.054518] Key type ._llcrypt unregistered [ 1329.952659] Key type ._llcrypt registered [ 1329.954845] Key type .llcrypt registered [ 1330.132880] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1330.151208] alg: No test for adler32 (adler32-zlib) [ 1331.057922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1331.110933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1331.134657] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 1331.741472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1331.760949] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1332.368255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.202.42@tcp [ 1333.407765] LNetError: Unexpected error -22 connecting to 192.168.202.42@tcp at host 192.168.202.42:988 [ 1336.629479] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 03:25:27 (1780385127) [ 1336.770653] LNet: 73059:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1336.774952] LNetError: 73059:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1336.780357] LNet: Removed LNI 192.168.202.42@tcp [ 1336.791559] LNet: Removed LNI 10.1.2.3@tcp [ 1337.165627] Key type .llcrypt unregistered [ 1337.168853] Key type ._llcrypt unregistered [ 1341.977816] Key type ._llcrypt registered [ 1341.980123] Key type .llcrypt registered [ 1342.141185] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1342.154289] alg: No test for adler32 (adler32-zlib) [ 1343.085124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1343.174583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1343.204983] LNet: Added LNI 192.168.202.42@tcp1 [8/256/0/180] [ 1343.937447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1343.956580] LNet: Added LNI 192.168.202.42@tcp2 [8/256/0/180] [ 1344.949874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.202.42@tcp1 --nid 192.168.202.42@tcp2 [ 1348.835745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.42@tcp1 192.168.202.42@tcp2 [ 1348.855808] LNetError: 73894:0:(lib-move.c:2007:lnet_handle_find_routed_path()) Unable to send message from 192.168.202.42@tcp1 to 192.168.202.42@tcp2 - Route table may be misconfigured [ 1348.880273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.42@tcp2 192.168.202.42@tcp1 [ 1354.406659] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 03:25:44 (1780385144) [ 1354.485488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1354.512772] LNet: 74459:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1354.521873] LNetError: 74459:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1355.551324] LNet: Removed LNI 192.168.202.42@tcp1 [ 1355.555741] LNetError: 74459:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1356.581736] LNet: Removed LNI 192.168.202.42@tcp2 [ 1356.622868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1356.692496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1356.730622] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 1357.334739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1357.346784] LNet: Added LNI 192.168.202.42@tcp1 [8/256/0/180] [ 1357.982716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.42@tcp [ 1358.013541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.42@tcp 192.168.202.42@tcp [ 1358.040788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.42@tcp1 192.168.202.42@tcp1 [ 1358.069586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.202.42@tcp -d 192.168.202.42@tcp -e network_timeout [ 1358.101495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.202.42@tcp -d 192.168.202.42@tcp1 -e network_timeout [ 1358.130455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.202.42@tcp1 -d 192.168.202.42@tcp -e network_timeout [ 1358.157972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.202.42@tcp1 -d 192.168.202.42@tcp1 -e network_timeout [ 1358.192387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.42@tcp [ 1358.209770] LNet: There was an unexpected network error while writing to 192.168.202.42: rc = -22 [ 1358.225299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1358.288308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1358.358645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1358.687158] LNet: 1 local NIs in recovery (showing 1): 192.168.202.42@tcp [ 1363.463914] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 03:25:53 (1780385153) [ 1363.535490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1363.560100] LNet: 75321:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1363.565315] LNetError: 75321:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1364.575162] LNet: Removed LNI 192.168.202.42@tcp [ 1365.605299] LNet: Removed LNI 192.168.202.42@tcp1 [ 1365.657976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1365.770731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1366.067100] LNet: 75367:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1366.362317] Key type .llcrypt unregistered [ 1366.363583] Key type ._llcrypt unregistered [ 1375.081301] Key type ._llcrypt registered [ 1375.083738] Key type .llcrypt registered [ 1376.094593] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 03:26:06 (1780385166) [ 1376.331984] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1376.342883] alg: No test for adler32 (adler32-zlib) [ 1377.241945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1378.589101] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1380.453649] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 03:26:10 (1780385170) [ 1380.523439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1380.555110] LNet: 76457:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1380.582839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1380.647001] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1380.685312] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 1381.428424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1381.446451] LNet: Added LNI 192.168.202.42@tcp1 [8/256/0/180] [ 1382.144931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.42@tcp [ 1382.177449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.42@tcp1 [ 1382.213966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.42@tcp1 [ 1386.821030] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 03:26:17 (1780385177) [ 1388.172943] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg242-server [ 1389.545423] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 03:26:19 (1780385179) [ 1391.048332] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg242-server [ 1392.468381] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 03:26:22 (1780385182) [ 1394.002598] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg242-server [ 1395.524700] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 03:26:25 (1780385185) [ 1397.326616] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg242-server [ 1398.759753] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 03:26:28 (1780385188) [ 1400.253815] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg242-server [ 1401.639053] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 03:26:31 (1780385191) [ 1402.896833] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg242-server [ 1404.568218] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 03:26:34 (1780385194) [ 1405.850615] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg242-server [ 1406.958261] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 03:26:37 (1780385197) [ 1408.421551] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg242-server [ 1409.855788] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 03:26:40 (1780385200) [ 1411.083895] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg242-server [ 1412.317412] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 03:26:42 (1780385202) [ 1412.408302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1412.421528] LNet: 80342:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1412.433053] LNetError: 80342:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1412.440444] LNet: Removed LNI 192.168.202.42@tcp [ 1413.480254] LNet: Removed LNI 192.168.202.42@tcp1 [ 1413.502841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1413.555562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1413.593943] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 1414.208930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1414.288741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.42@tcp [ 1414.445776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1414.466406] LNet: 80489:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1414.472494] LNetError: 80489:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1414.476172] LNetError: 80489:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1414.489907] LNet: Removed LNI 192.168.202.42@tcp [ 1414.503238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1414.548734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1414.574951] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 1415.174929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.42@tcp [ 1415.382492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1416.425899] LNet: Removed LNI 192.168.202.42@tcp [ 1416.442780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1416.491383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1416.519245] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 1417.084875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1417.160484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.42@tcp [ 1417.285596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1417.312598] LNet: 80782:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1417.322136] LNet: 80782:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1417.331225] LNetError: 80782:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1417.341556] LNetError: 80782:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1417.365358] LNet: Removed LNI 192.168.202.42@tcp [ 1417.399132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1417.447310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1417.467589] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 1418.054527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.42@tcp [ 1418.204631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1419.303107] LNet: Removed LNI 192.168.202.42@tcp [ 1419.329448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1419.421273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1419.455341] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 1420.085768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1420.149175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.42@tcp [ 1420.424758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1420.448964] LNetError: 81074:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1420.451835] LNetError: 81074:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1421.476909] LNet: Removed LNI 192.168.202.42@tcp [ 1421.536355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1421.594289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1421.622376] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 1422.222491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.42@tcp [ 1422.391370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1422.409727] LNet: 81219:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1422.414909] LNet: 81219:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 1423.480790] LNet: Removed LNI 192.168.202.42@tcp [ 1423.508328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1423.557061] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1423.608929] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 1424.317264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1424.367544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.42@tcp [ 1424.516291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1424.530855] LNetError: 81367:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1424.535096] LNetError: 81367:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1425.575808] LNet: Removed LNI 192.168.202.42@tcp [ 1425.603988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1425.688057] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1425.706797] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 1426.305524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.42@tcp [ 1427.654295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1428.715899] LNet: Removed LNI 192.168.202.42@tcp [ 1428.744414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1428.801993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1428.840923] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 1429.596962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1429.672542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.42@tcp [ 1429.867988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1430.954937] LNet: Removed LNI 192.168.202.42@tcp [ 1431.006773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1431.091587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1431.124963] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 1431.656488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.42@tcp [ 1431.887692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1431.909713] LNet: 81814:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1431.914265] LNet: 81814:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 1432.936774] LNet: Removed LNI 192.168.202.42@tcp [ 1432.959433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1433.054215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1433.105225] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 1433.744791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1433.800652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.42@tcp [ 1435.179855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1435.210348] LNetError: 81970:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1435.218569] LNetError: 81970:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 1435.259966] LNet: Removed LNI 192.168.202.42@tcp [ 1435.291439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1435.359571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1435.396929] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 1436.207705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.42@tcp [ 1436.415326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1437.480881] LNet: Removed LNI 192.168.202.42@tcp [ 1437.529946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1437.589847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1437.618280] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 1438.456856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1438.508537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1438.535975] LNet: Removed LNI 192.168.202.42@tcp [ 1438.547821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1438.598972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1438.621113] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 1439.375477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1445.603379] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 03:27:15 (1780385235) [ 1445.876809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1445.909684] LNet: Removed LNI 192.168.202.42@tcp [ 1445.937580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1445.993875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1446.010691] LNet: Added LNI 192.168.202.42@tcp231 [8/256/0/180] [ 1446.999545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1447.018298] LNet: Removed LNI 192.168.202.42@tcp231 [ 1447.031631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1447.088838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1447.119277] LNet: Added LNI 192.168.202.42@tcp231 [8/256/0/180] [ 1447.123594] LNet: 83099:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1447.477530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1447.513388] LNet: Removed LNI 192.168.202.42@tcp231 [ 1447.536556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1447.569234] LNet: Added LNI 192.168.202.42@tcp231 [8/256/0/180] [ 1448.482122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1448.501107] LNet: 83243:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1448.505180] LNet: 83243:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 5 previous similar messages [ 1448.516490] LNet: Removed LNI 192.168.202.42@tcp231 [ 1448.535640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1448.597069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1448.632821] LNet: Added LNI 192.168.202.42@tcp231 [8/256/0/0] [ 1448.641923] LNet: 83251:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1449.050770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1449.085217] LNet: Removed LNI 192.168.202.42@tcp231 [ 1449.103605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1449.125904] LNet: Added LNI 192.168.202.42@tcp231 [8/256/0/0] [ 1450.088667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1450.113364] LNet: Removed LNI 192.168.202.42@tcp231 [ 1450.139702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1450.211971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1450.238729] LNet: Added LNI 192.168.202.42@tcp231 [8/256/0/60] [ 1450.251129] LNet: 83402:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1450.561618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1450.588982] LNet: Removed LNI 192.168.202.42@tcp231 [ 1450.604902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1450.631532] LNet: Added LNI 192.168.202.42@tcp231 [8/256/0/60] [ 1451.558778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1451.590134] LNetError: 83545:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1451.597092] LNetError: 83545:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 6 previous similar messages [ 1451.607411] LNet: Removed LNI 192.168.202.42@tcp231 [ 1451.624263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1451.679963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1451.695655] LNet: Added LNI 192.168.202.42@tcp231 [8/256/0/180] [ 1452.187124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1452.232322] LNet: Removed LNI 192.168.202.42@tcp231 [ 1452.247878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1452.273699] LNet: Added LNI 192.168.202.42@tcp231 [8/256/0/180] [ 1457.876508] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 03:27:28 (1780385248) [ 1458.080535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1458.110190] LNet: Removed LNI 192.168.202.42@tcp231 [ 1458.137876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1458.208620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1458.247525] LNet: Added LNI 192.168.202.42@tcp232 [8/256/0/180] [ 1458.792468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1458.819657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1458.847679] LNet: Removed LNI 192.168.202.42@tcp232 [ 1458.856331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1458.867430] LNet: Added LNI 192.168.202.42@tcp232 [8/256/0/180] [ 1459.538305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1459.571465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1459.612541] LNet: Removed LNI 192.168.202.42@tcp232 [ 1459.632990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1459.688656] LNet: Added LNI 192.168.202.42@tcp232 [8/256/0/180] [ 1460.256812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1460.279148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1460.326108] LNet: Removed LNI 192.168.202.42@tcp232 [ 1460.341351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1460.366160] LNet: Added LNI 192.168.202.42@tcp232 [8/256/0/180] [ 1461.001014] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1461.037387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1461.068719] LNet: Removed LNI 192.168.202.42@tcp232 [ 1461.091562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1466.090970] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 03:27:36 (1780385256) [ 1467.010331] Key type .llcrypt unregistered [ 1467.011691] Key type ._llcrypt unregistered [ 1471.790854] Key type ._llcrypt registered [ 1471.793554] Key type .llcrypt registered [ 1471.959518] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1471.982336] alg: No test for adler32 (adler32-zlib) [ 1472.853394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1472.879447] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 1478.235604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.142@tcp [ 1483.855567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1483.899688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1483.914655] LNet: Added LNI 192.168.202.42@tcp1 [8/256/0/180] [ 1484.882260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.42@tcp -d 192.168.202.142@tcp -m GET -e local_interrupt -r 2 [ 1484.911742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.42@tcp -d 192.168.202.142@tcp1 -m GET -e local_interrupt -r 2 [ 1484.941617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.42@tcp -d 192.168.202.42@tcp -m GET -e local_interrupt -r 2 [ 1484.973826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.42@tcp -d 192.168.202.42@tcp1 -m GET -e local_interrupt -r 2 [ 1485.013516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.42@tcp1 -d 192.168.202.142@tcp -m GET -e local_interrupt -r 2 [ 1485.046427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.42@tcp1 -d 192.168.202.142@tcp1 -m GET -e local_interrupt -r 2 [ 1485.084674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.42@tcp1 -d 192.168.202.42@tcp -m GET -e local_interrupt -r 2 [ 1485.116233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.42@tcp1 -d 192.168.202.42@tcp1 -m GET -e local_interrupt -r 2 [ 1485.146789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.42@tcp 192.168.202.142@tcp [ 1485.176456] LNet: There was an unexpected network error while writing to 192.168.202.142: rc = -22 [ 1485.182404] LNet: 1 local NIs in recovery (showing 1): 192.168.202.42@tcp [ 1485.325433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1485.347300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1485.386975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1485.409952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1485.437709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1485.459299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1485.498530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1485.520546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1485.535201] LNet: 86052:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1485.546856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1485.569662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1485.593557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1485.612783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1485.645822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1485.669401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1486.076139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.42@tcp -d 192.168.202.142@tcp -m GET -e local_dropped -r 2 [ 1486.107469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.42@tcp -d 192.168.202.142@tcp1 -m GET -e local_dropped -r 2 [ 1486.135500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.42@tcp -d 192.168.202.42@tcp -m GET -e local_dropped -r 2 [ 1486.163223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.42@tcp -d 192.168.202.42@tcp1 -m GET -e local_dropped -r 2 [ 1486.192033] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.42@tcp1 -d 192.168.202.142@tcp -m GET -e local_dropped -r 2 [ 1486.221209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.42@tcp1 -d 192.168.202.142@tcp1 -m GET -e local_dropped -r 2 [ 1486.255182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.42@tcp1 -d 192.168.202.42@tcp -m GET -e local_dropped -r 2 [ 1486.283679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.42@tcp1 -d 192.168.202.42@tcp1 -m GET -e local_dropped -r 2 [ 1486.308274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.42@tcp 192.168.202.142@tcp [ 1486.322915] LNet: There was an unexpected network error while writing to 192.168.202.142: rc = -22 [ 1486.471463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1486.497871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1486.517775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1486.538374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1486.560213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1486.581872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1486.605273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1486.629181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1486.646615] LNet: 86172:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1486.662069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1486.688739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1486.710242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1486.734860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1486.764241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1486.796841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1487.214709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.42@tcp -d 192.168.202.142@tcp -m GET -e local_aborted -r 2 [ 1487.244851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.42@tcp -d 192.168.202.142@tcp1 -m GET -e local_aborted -r 2 [ 1487.271239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.42@tcp -d 192.168.202.42@tcp -m GET -e local_aborted -r 2 [ 1487.296310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.42@tcp -d 192.168.202.42@tcp1 -m GET -e local_aborted -r 2 [ 1487.322708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.42@tcp1 -d 192.168.202.142@tcp -m GET -e local_aborted -r 2 [ 1487.354479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.42@tcp1 -d 192.168.202.142@tcp1 -m GET -e local_aborted -r 2 [ 1487.382101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.42@tcp1 -d 192.168.202.42@tcp -m GET -e local_aborted -r 2 [ 1487.404235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.42@tcp1 -d 192.168.202.42@tcp1 -m GET -e local_aborted -r 2 [ 1487.438697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.42@tcp 192.168.202.142@tcp [ 1487.457488] LNet: There was an unexpected network error while writing to 192.168.202.142: rc = -22 [ 1487.692508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1487.723708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1487.748703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1487.773105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1487.801470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1487.826200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1487.851465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1487.875329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1487.891682] LNet: 86293:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1487.908465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1487.942653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1487.968434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1488.003660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1488.035877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1488.064794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1488.498208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.42@tcp -d 192.168.202.142@tcp -m GET -e local_no_route -r 2 [ 1488.530484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.42@tcp -d 192.168.202.142@tcp1 -m GET -e local_no_route -r 2 [ 1488.561759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.42@tcp -d 192.168.202.42@tcp -m GET -e local_no_route -r 2 [ 1488.587575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.42@tcp -d 192.168.202.42@tcp1 -m GET -e local_no_route -r 2 [ 1488.617790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.42@tcp1 -d 192.168.202.142@tcp -m GET -e local_no_route -r 2 [ 1488.643260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.42@tcp1 -d 192.168.202.142@tcp1 -m GET -e local_no_route -r 2 [ 1488.673672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.42@tcp1 -d 192.168.202.42@tcp -m GET -e local_no_route -r 2 [ 1488.714426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.42@tcp1 -d 192.168.202.42@tcp1 -m GET -e local_no_route -r 2 [ 1488.748412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.42@tcp 192.168.202.142@tcp [ 1488.956269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1488.974738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1488.996853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1489.030811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1489.058196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1489.097262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1489.137717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1489.162750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1489.190109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1489.222857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1489.246032] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1489.276639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1489.311819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1489.339337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1489.885674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.42@tcp -d 192.168.202.142@tcp -m GET -e local_timeout -r 2 [ 1489.913519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.42@tcp -d 192.168.202.142@tcp1 -m GET -e local_timeout -r 2 [ 1489.961989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.42@tcp -d 192.168.202.42@tcp -m GET -e local_timeout -r 2 [ 1489.987269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.42@tcp -d 192.168.202.42@tcp1 -m GET -e local_timeout -r 2 [ 1490.019785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.42@tcp1 -d 192.168.202.142@tcp -m GET -e local_timeout -r 2 [ 1490.060873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.42@tcp1 -d 192.168.202.142@tcp1 -m GET -e local_timeout -r 2 [ 1490.091346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.42@tcp1 -d 192.168.202.42@tcp -m GET -e local_timeout -r 2 [ 1490.114840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.42@tcp1 -d 192.168.202.42@tcp1 -m GET -e local_timeout -r 2 [ 1490.149207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.42@tcp 192.168.202.142@tcp [ 1490.168153] LNet: There was an unexpected network error while writing to 192.168.202.142: rc = -22 [ 1490.173802] LNet: Skipped 1 previous similar message [ 1490.416510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1490.452921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1490.487664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1490.513502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1490.537627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1490.564235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1490.579844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1490.608339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1490.633972] LNet: 86533:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1490.643130] LNet: 86533:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 1490.669816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1490.703776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1490.730543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1490.759642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1490.792272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1490.818771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1491.233847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.42@tcp -d 192.168.202.142@tcp -m GET -e remote_dropped -r 2 [ 1491.282437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.42@tcp -d 192.168.202.142@tcp1 -m GET -e remote_dropped -r 2 [ 1491.319799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.42@tcp -d 192.168.202.42@tcp -m GET -e remote_dropped -r 2 [ 1491.342465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.42@tcp -d 192.168.202.42@tcp1 -m GET -e remote_dropped -r 2 [ 1491.382472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.42@tcp1 -d 192.168.202.142@tcp -m GET -e remote_dropped -r 2 [ 1491.414102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.42@tcp1 -d 192.168.202.142@tcp1 -m GET -e remote_dropped -r 2 [ 1491.449617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.42@tcp1 -d 192.168.202.42@tcp -m GET -e remote_dropped -r 2 [ 1491.480240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.42@tcp1 -d 192.168.202.42@tcp1 -m GET -e remote_dropped -r 2 [ 1491.510430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.42@tcp 192.168.202.142@tcp [ 1491.526471] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.142@tcp [ 1491.687829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1491.710951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1491.739883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1491.762069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1491.788655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1491.816941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1491.842719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1491.862296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1491.886375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1491.907346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1491.929978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1491.953309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1491.977462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1492.002964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1495.740757] LNet: 86727:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1495.745761] LNetError: 86727:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1495.749916] LNet: Removed LNI 192.168.202.42@tcp [ 1495.762950] LNet: Removed LNI 192.168.202.42@tcp1 [ 1496.117903] Key type .llcrypt unregistered [ 1496.120371] Key type ._llcrypt unregistered [ 1506.446134] Key type ._llcrypt registered [ 1506.447877] Key type .llcrypt registered [ 1507.386218] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 03:28:17 (1780385297) [ 1507.671100] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1507.679621] alg: No test for adler32 (adler32-zlib) [ 1508.539691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1508.573590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1508.587742] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 1509.069333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1509.084281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.202.42@tcp [ 1509.098236] LNetError: 87705:0:(api-ni.c:4350:lnet_ni_set_healthv()) manually adding local NI 192.168.202.42@tcp to recovery [ 1509.159603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.202.42@tcp [ 1509.232452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1513.345814] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 03:28:23 (1780385303) [ 1513.476721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1513.496592] LNet: 88294:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1513.502256] LNetError: 88294:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1513.517511] LNet: Removed LNI 192.168.202.42@tcp [ 1513.530412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1513.564919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1513.584270] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 1514.065957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1514.080613] LNet: Added LNI 192.168.202.42@tcp1 [8/256/0/180] [ 1514.675465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.202.42@tcp [ 1514.691289] LNetError: 88560:0:(api-ni.c:4350:lnet_ni_set_healthv()) manually adding local NI 192.168.202.42@tcp to recovery [ 1515.616531] LNet: 1 local NIs in recovery (showing 1): 192.168.202.42@tcp [ 1515.621715] LNet: There was an unexpected network error while writing to 192.168.202.42: rc = -22 [ 1515.936733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.202.42@tcp [ 1522.698416] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 03:28:33 (1780385313) [ 1524.073857] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg242-server [ 1525.449514] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 03:28:35 (1780385315) [ 1526.788463] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg242-server [ 1528.336229] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 03:28:38 (1780385318) [ 1529.494922] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg242-server [ 1530.747371] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 03:28:41 (1780385321) [ 1530.883291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1530.894816] LNet: 90190:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1530.901053] LNetError: 90190:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1530.905676] LNet: Removed LNI 192.168.202.42@tcp [ 1530.918342] LNet: Removed LNI 192.168.202.42@tcp1 [ 1530.931735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1530.966543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1530.986220] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 1531.568777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1531.625146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1531.664751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1531.718575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1531.753769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1531.783626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1535.975593] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 03:28:46 (1780385326) [ 1536.142327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1536.157801] LNet: 90915:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1536.164775] LNetError: 90915:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1536.168821] LNetError: 90915:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1536.176979] LNet: Removed LNI 192.168.202.42@tcp [ 1536.189501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1536.224626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1536.238071] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 1540.979624] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 03:28:51 (1780385331) [ 1541.089481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1541.112115] LNet: 91615:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1541.118314] LNetError: 91615:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1541.129949] LNet: Removed LNI 192.168.202.42@tcp [ 1541.148995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1541.194358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1541.209289] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 1546.279740] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 03:28:56 (1780385336) [ 1547.465850] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 1548.656703] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 03:28:59 (1780385339) [ 1548.911379] LNet: 92652:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1548.913744] LNetError: 92652:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1548.921936] LNet: Removed LNI 192.168.202.42@tcp [ 1549.265771] Key type .llcrypt unregistered [ 1549.267774] Key type ._llcrypt unregistered [ 1553.993079] Key type ._llcrypt registered [ 1553.994479] Key type .llcrypt registered [ 1554.100645] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1554.110390] alg: No test for adler32 (adler32-zlib) [ 1554.948392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1554.978726] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 1560.957843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.142@tcp [ 1567.469300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.202.142@tcp [ 1568.337262] LNet: 93069:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1568.364161] LNetError: 93069:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1568.414499] LNet: Removed LNI 192.168.202.42@tcp [ 1569.457466] Key type .llcrypt unregistered [ 1569.460587] Key type ._llcrypt unregistered [ 1587.505837] Key type ._llcrypt registered [ 1587.513443] Key type .llcrypt registered [ 1590.648764] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 1593.126750] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 1596.483439] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 03:29:46 (1780385386) [ 1598.743889] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 [ 1600.813926] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 03:29:50 (1780385390) [ 1603.235781] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg242-server [ 1604.911404] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 03:29:54 (1780385394) [ 1606.685875] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg242-server [ 1608.329460] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 03:29:58 (1780385398) [ 1610.494054] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1610.524642] alg: No test for adler32 (adler32-zlib) [ 1611.691991] Lustre: Lustre: Build Version: 2.17.53_57_g7d1c020 [ 1611.986803] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 1613.775160] Key type lgssc registered [ 1615.602453] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1635.664332] Key type lgssc unregistered [ 1635.943811] LNet: 96246:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1635.949525] LNetError: 96246:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1635.966880] LNet: Removed LNI 192.168.202.42@tcp [ 1636.524881] Key type .llcrypt unregistered [ 1636.526793] Key type ._llcrypt unregistered [ 1650.077375] Key type ._llcrypt registered [ 1650.079632] Key type .llcrypt registered [ 1651.096777] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 03:30:41 (1780385441) [ 1651.423677] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1651.435650] alg: No test for adler32 (adler32-zlib) [ 1652.305247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1652.380889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1652.428717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1652.468908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1652.505155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1657.562521] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 03:30:47 (1780385447) [ 1657.668700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1657.700230] LNet: 97657:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1657.741571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1657.773533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1657.788659] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 1658.478225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1658.494710] LNet: 97805:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1658.499420] LNetError: 97805:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1658.511911] LNet: Removed LNI 192.168.202.42@tcp [ 1658.527070] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1658.568535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1658.586387] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 1658.975424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1659.008388] LNet: Removed LNI 192.168.202.42@tcp [ 1659.036280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1659.091147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1659.116206] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 1664.304424] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 03:30:54 (1780385454) [ 1664.515860] LNet: 98423:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1664.524861] LNet: 98423:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1664.537528] LNetError: 98423:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1664.548800] LNetError: 98423:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1664.561068] LNet: Removed LNI 192.168.202.42@tcp [ 1665.126327] Key type .llcrypt unregistered [ 1665.130216] Key type ._llcrypt unregistered [ 1670.475404] Key type ._llcrypt registered [ 1670.477737] Key type .llcrypt registered [ 1670.688936] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1670.717364] alg: No test for adler32 (adler32-zlib) [ 1671.645983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1671.728573] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 1671.732118] LNet: *** cfs_fail_loc=e003, val=100*** [ 1672.446501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.42@tcp [ 1672.462510] LNet: *** cfs_fail_loc=e003, val=100*** [ 1672.464960] LNet: Skipped 2 previous similar messages [ 1672.471985] Ping reply received with mapping: [ 1672.475416] found metadata reply NI[192.168.202.42@tcp] [ 1672.478841] 2a ca a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 1672.481766] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 1672.485356] 5a 5a 5a 5a 5a 5a 5a 5a [ 1672.593154] LNet: 98820:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1672.599886] LNetError: 98820:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1673.642215] LNet: Removed LNI 192.168.202.42@tcp [ 1674.545367] Key type .llcrypt unregistered [ 1674.547090] Key type ._llcrypt unregistered [ 1685.879737] Key type ._llcrypt registered [ 1685.888477] Key type .llcrypt registered [ 1687.277889] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 03:31:17 (1780385477) [ 1687.885242] Key type .llcrypt unregistered [ 1687.887755] Key type ._llcrypt unregistered [ 1694.850856] Key type ._llcrypt registered [ 1694.853450] Key type .llcrypt registered [ 1695.059537] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1695.079830] alg: No test for adler32 (adler32-zlib) [ 1696.329986] LNetError: 99866:0:(api-ni.c:2729:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 1696.691454] Key type .llcrypt unregistered [ 1696.692916] Key type ._llcrypt unregistered [ 1701.677270] Key type ._llcrypt registered [ 1701.679645] Key type .llcrypt registered [ 1702.914315] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 03:31:32 (1780385492) [ 1703.333305] Key type .llcrypt unregistered [ 1703.334950] Key type ._llcrypt unregistered [ 1708.476997] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1708.480996] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1710.965731] Key type ._llcrypt registered [ 1710.973443] Key type .llcrypt registered [ 1711.199734] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1711.209180] alg: No test for adler32 (adler32-zlib) [ 1712.252329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1712.343994] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1721.023527] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 03:31:51 (1780385511) [ 1721.140765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1721.181133] LNet: 101641:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1721.199346] LNetError: 101641:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1721.231232] LNet: Removed LNI 10.1.2.3@tcp [ 1721.290639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1721.981644] Key type .llcrypt unregistered [ 1721.985875] Key type ._llcrypt unregistered [ 1728.110794] Key type ._llcrypt registered [ 1728.113294] Key type .llcrypt registered [ 1728.273826] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1728.295585] alg: No test for adler32 (adler32-zlib) [ 1729.294835] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 1729.382339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.42@tcp [ 1729.570913] LNet: 101934:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1729.576976] LNetError: 101934:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1730.681245] LNet: Removed LNI 192.168.202.42@tcp [ 1731.461587] Key type .llcrypt unregistered [ 1731.465069] Key type ._llcrypt unregistered [ 1744.902623] Key type ._llcrypt registered [ 1744.904910] Key type .llcrypt registered [ 1746.392094] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 03:32:16 (1780385536) [ 1747.066278] Key type .llcrypt unregistered [ 1747.069788] Key type ._llcrypt unregistered [ 1753.721526] Key type ._llcrypt registered [ 1753.727482] Key type .llcrypt registered [ 1753.977573] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1754.005290] alg: No test for adler32 (adler32-zlib) [ 1760.826825] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 03:32:30 (1780385550) [ 1761.001678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1761.027446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1761.076680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1761.103389] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 1761.659883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1761.695160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 1761.720280] LNet: Removed LNI 192.168.202.42@tcp [ 1762.079317] LNet: 103678:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1762.429668] Key type .llcrypt unregistered [ 1762.431695] Key type ._llcrypt unregistered [ 1773.447875] Key type ._llcrypt registered [ 1773.450483] Key type .llcrypt registered [ 1774.821871] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 03:32:44 (1780385564) [ 1775.294809] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1775.312728] alg: No test for adler32 (adler32-zlib) [ 1776.341413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1776.394545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1776.426670] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 1777.237729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.42@tcp [ 1777.290843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.202.42@tcp [ 1777.322623] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 1777.327488] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 1777.335054] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1777, 0, 0 [ 1777.342708] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 1777.347501] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 1777.353808] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 1777.361169] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1777, 0, 0 [ 1777.365776] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 1777.368810] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1777.373114] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 1777.377113] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1777, 0, 1827 [ 1777.389315] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 1777.390466] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1777.394978] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 1777.402606] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1777, 0, 1827 [ 1777.410612] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 1783.963775] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 03:32:53 (1780385573) [ 1784.617679] LNet: 105217:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1784.627120] LNetError: 105217:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1784.644616] LNet: Removed LNI 192.168.202.42@tcp [ 1785.370545] Key type .llcrypt unregistered [ 1785.374384] Key type ._llcrypt unregistered [ 1793.582486] Key type ._llcrypt registered [ 1793.585900] Key type .llcrypt registered [ 1793.861513] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1793.882800] alg: No test for adler32 (adler32-zlib) [ 1794.927616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1794.992111] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 1802.533880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.142@tcp [ 1811.095909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1811.156831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1811.176409] LNet: Added LNI 192.168.202.42@tcp1 [8/256/0/180] [ 1812.130788] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1812.260610] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1812.273607] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1812.280712] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1814.682207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 1814.697236] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 1826.040368] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 03:33:35 (1780385615) [ 1826.195616] LNet: 106715:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1826.199715] LNetError: 106715:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1826.203164] LNet: Removed LNI 192.168.202.42@tcp [ 1827.295217] LNet: Removed LNI 192.168.202.42@tcp1 [ 1827.297224] LNetError: 106715:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1827.307649] LNetError: 106715:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1827.329659] LNet: Removed LNI 10.1.2.3@tcp99 [ 1828.191922] Key type .llcrypt unregistered [ 1828.194946] Key type ._llcrypt unregistered [ 1835.974463] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1835.980603] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1838.556589] Key type ._llcrypt registered [ 1838.559548] Key type .llcrypt registered [ 1838.766615] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1838.799855] alg: No test for adler32 (adler32-zlib) [ 1839.816723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1839.867769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1839.887658] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 1840.733799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1840.790886] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1843.000988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 1843.069372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1843.108948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 1843.127829] LNetError: 107469:0:(peer.c:2175:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 1843.163671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1843.271452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.42@tcp [ 1843.964807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1844.077640] LNet: 107489:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1844.083947] LNetError: 107489:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1845.087146] LNet: Removed LNI 192.168.202.42@tcp [ 1845.093470] LNetError: 107489:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1845.125064] LNet: Removed LNI 10.1.2.3@tcp [ 1845.879408] Key type .llcrypt unregistered [ 1845.880989] Key type ._llcrypt unregistered [ 1859.432882] Key type ._llcrypt registered [ 1859.434488] Key type .llcrypt registered [ 1860.843904] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 03:34:10 (1780385650) [ 1861.395526] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1861.412785] alg: No test for adler32 (adler32-zlib) [ 1862.375758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1862.423386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1862.446583] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 1863.342951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.42@tcp [ 1863.451812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg242-client@tcp [ 1870.683977] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 03:34:20 (1780385660) [ 1870.819215] LNet: 109016:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1870.824624] LNetError: 109016:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1870.844511] LNet: Removed LNI 192.168.202.42@tcp [ 1871.446418] Key type .llcrypt unregistered [ 1871.448244] Key type ._llcrypt unregistered [ 1879.012207] Key type ._llcrypt registered [ 1879.017238] Key type .llcrypt registered [ 1879.299241] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1879.330442] alg: No test for adler32 (adler32-zlib) [ 1880.347486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 1880.384893] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 1880.417737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1886.628477] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 03:34:36 (1780385676) [ 1888.006126] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 1890.035440] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 03:34:39 (1780385679) [ 1891.878528] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 1893.762495] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 03:34:43 (1780385683) [ 1893.931417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1893.989037] LNet: 110442:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1894.010766] LNetError: 110442:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1894.031892] LNet: Removed LNI 192.168.202.42@tcp [ 1894.048277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1894.103817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1894.147079] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 1895.013589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 1895.059617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 1902.267473] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 03:34:52 (1780385692) [ 1902.469254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1902.495972] LNet: 111148:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1902.504208] LNetError: 111148:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1902.539707] LNet: Removed LNI 192.168.202.42@tcp [ 1902.601457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1902.658525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 1902.684557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1903.371265] Key type .llcrypt unregistered [ 1903.373511] Key type ._llcrypt unregistered [ 1914.312929] Key type ._llcrypt registered [ 1914.318275] Key type .llcrypt registered [ 1915.668618] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 03:35:05 (1780385705) [ 1916.015046] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1916.052584] alg: No test for adler32 (adler32-zlib) [ 1917.006613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1917.064612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1917.100346] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 1917.908682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1917.978584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.42@tcp [ 1923.599412] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 03:35:13 (1780385713) [ 1923.732690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1923.773308] LNet: 112704:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1923.782261] LNetError: 112704:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1923.803127] LNet: Removed LNI 192.168.202.42@tcp [ 1923.835934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1923.881295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 1923.899309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 1929.445456] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 03:35:19 (1780385719) [ 1929.593990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1929.622670] LNet: 113287:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1929.653906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1929.707288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 1929.965501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1929.989954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1930.028742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 1930.384863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1930.413767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1930.463107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1930.492770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1930.532678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1930.770246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1930.797297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1930.837352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 1931.072923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1931.096728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1936.930982] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 03:35:26 (1780385726) [ 1937.145176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1937.179617] LNet: 113922:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1937.183404] LNet: 113922:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1937.219931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1937.288834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1937.312767] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 1937.937740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1937.981130] LNet: Added LNI 192.168.202.42@tcp2 [8/256/0/180] [ 1938.806261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 1938.856175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.42@tcp [ 1939.064240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1939.080649] LNetError: 114201:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1940.127562] LNet: Removed LNI 192.168.202.42@tcp [ 1940.131296] LNetError: 114201:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1940.149855] LNet: Removed LNI 192.168.202.42@tcp2 [ 1940.181554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1940.258717] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 1940.263904] LNet: Added LNI 192.168.202.42@tcp2 [8/256/0/180] [ 1940.272096] LNet: 114207:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1940.902952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.42@tcp [ 1941.126812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1941.154511] LNet: 114244:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1941.161036] LNet: 114244:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1942.175242] LNet: Removed LNI 192.168.202.42@tcp [ 1942.180734] LNetError: 114244:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1942.186849] LNetError: 114244:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1942.219785] LNet: Removed LNI 192.168.202.42@tcp2 [ 1942.239621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1942.294587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1942.310465] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 1943.100586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1943.114083] LNet: Added LNI 192.168.202.42@tcp2 [8/256/0/180] [ 1943.781918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.42@tcp [ 1943.813500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 1943.858477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.42@tcp [ 1944.043617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1945.119225] LNet: Removed LNI 192.168.202.42@tcp [ 1946.153617] LNet: Removed LNI 192.168.202.42@tcp2 [ 1946.220857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1946.271391] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 1946.278589] LNet: Added LNI 192.168.202.42@tcp2 [8/256/0/180] [ 1946.284490] LNet: 114535:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1946.720565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.42@tcp [ 1946.937713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1946.960762] LNet: 114572:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1946.967512] LNet: 114572:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1946.972699] LNetError: 114572:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1946.975406] LNetError: 114572:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1946.985085] LNet: Removed LNI 192.168.202.42@tcp [ 1948.005714] LNet: Removed LNI 192.168.202.42@tcp2 [ 1948.036665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1948.102876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1948.121977] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 1948.648730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1948.665442] LNet: Added LNI 192.168.202.42@tcp2 [8/256/0/180] [ 1949.469519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1949.538306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.42@tcp [ 1949.804764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1950.879137] LNet: Removed LNI 192.168.202.42@tcp [ 1950.914698] LNet: Removed LNI 192.168.202.42@tcp2 [ 1950.973842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1951.050662] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 1951.064111] LNet: Added LNI 192.168.202.42@tcp2 [8/256/0/180] [ 1951.069865] LNet: 114858:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1951.644544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.42@tcp [ 1951.877893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1952.927404] LNet: Removed LNI 192.168.202.42@tcp [ 1952.941650] LNet: Removed LNI 192.168.202.42@tcp2 [ 1953.000617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1953.129782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1953.179899] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 1954.003888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1954.018960] LNet: Added LNI 192.168.202.42@tcp2 [8/256/0/180] [ 1955.006475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.42@tcp [ 1955.036720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 1955.090346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.42@tcp [ 1955.306521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1955.330624] LNet: 115179:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1955.344827] LNet: 115179:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 1955.358509] LNetError: 115179:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1955.372739] LNetError: 115179:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 1955.380774] LNet: Removed LNI 192.168.202.42@tcp [ 1956.462537] LNet: Removed LNI 192.168.202.42@tcp2 [ 1956.502613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1956.631873] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 1956.647319] LNet: Added LNI 192.168.202.42@tcp2 [8/256/0/180] [ 1956.667748] LNet: 115185:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1957.336992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.42@tcp [ 1957.783287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1957.812874] LNet: Removed LNI 192.168.202.42@tcp [ 1958.889735] LNet: Removed LNI 192.168.202.42@tcp2 [ 1958.945630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1959.052083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1959.094979] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 1960.098473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1960.123268] LNet: Added LNI 192.168.202.42@tcp2 [8/256/0/180] [ 1961.343904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 1961.398705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1961.467893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.42@tcp [ 1961.833994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1962.912911] LNet: Removed LNI 192.168.202.42@tcp [ 1962.939511] LNet: Removed LNI 192.168.202.42@tcp2 [ 1962.991961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1963.087368] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 1963.097201] LNet: Added LNI 192.168.202.42@tcp2 [8/256/0/180] [ 1963.101435] LNet: 115525:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1963.727974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.42@tcp [ 1964.145529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1965.215438] LNet: Removed LNI 192.168.202.42@tcp [ 1965.225959] LNet: Removed LNI 192.168.202.42@tcp2 [ 1965.290610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1965.365246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1965.404330] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 1966.107978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1966.132774] LNet: Added LNI 192.168.202.42@tcp2 [8/256/0/180] [ 1967.091395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.42@tcp [ 1967.157627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 1967.227412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 1967.309438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.42@tcp [ 1967.697783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1968.735759] LNet: Removed LNI 192.168.202.42@tcp [ 1969.767304] LNet: Removed LNI 192.168.202.42@tcp2 [ 1969.825023] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1969.934237] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 1969.950080] LNet: Added LNI 192.168.202.42@tcp2 [8/256/0/180] [ 1970.640598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.42@tcp [ 1976.949848] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 1978.264414] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 03:36:08 (1780385768) [ 1978.355790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1978.406485] LNet: 116638:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1978.415838] LNet: 116638:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 1978.443610] LNetError: 116638:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1978.453025] LNetError: 116638:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 9 previous similar messages [ 1978.467213] LNet: Removed LNI 192.168.202.42@tcp [ 1979.506074] LNet: Removed LNI 192.168.202.42@tcp2 [ 1979.556561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1979.942908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1979.988127] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 1980.518844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1980.537073] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1981.357469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1981.394498] LNet: Removed LNI 192.168.202.42@tcp [ 1981.405861] LNet: Removed LNI 10.1.2.3@tcp [ 1981.427345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1981.475811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 1981.501300] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 1981.508475] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1981.793587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 1981.814469] LNet: Removed LNI 192.168.202.42@tcp [ 1981.823956] LNet: Removed LNI 10.1.2.3@tcp [ 1988.593949] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 03:36:18 (1780385778) [ 1988.707274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1988.784087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1988.953419] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1989.147237] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1989.156245] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1989.182675] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1991.608184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1991.645690] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 1992.488669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1992.514553] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1994.870498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.42@tcp [ 1994.906086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 1994.966518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.202.42@tcp -m PUT -e local_timeout -r 1 [ 1995.025390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 1995.080031] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1995.083621] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1995.251760] LNetError: 117583:0:(lib-move.c:4710:lnet_parse()) 192.168.202.42@tcp, src 192.168.202.42@tcp: Dropping PUT (error -108 looking up sender) [ 1997.279521] LNet: Removed LNI 192.168.202.42@tcp [ 1997.292399] LNet: Removed LNI 10.1.2.3@tcp [ 1997.726347] Key type .llcrypt unregistered [ 1997.728325] Key type ._llcrypt unregistered [ 2008.060629] Key type ._llcrypt registered [ 2008.062231] Key type .llcrypt registered [ 2009.101400] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 03:36:39 (1780385799) [ 2009.375344] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2009.389515] alg: No test for adler32 (adler32-zlib) [ 2010.258859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2010.539295] LNet: 118894:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2011.062269] Key type .llcrypt unregistered [ 2011.063980] Key type ._llcrypt unregistered [ 2016.311272] Key type ._llcrypt registered [ 2016.314237] Key type .llcrypt registered [ 2016.464643] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2016.476843] alg: No test for adler32 (adler32-zlib) [ 2017.335221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2017.353559] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 2023.189223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.142@tcp [ 2023.346058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 2023.813392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.202.142@tcp [ 2024.022310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2024.062219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2024.091095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2024.136182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2024.168314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2024.199916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2024.253168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2024.288812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2024.305943] LNet: 119344:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2024.322350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2024.373232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2024.419380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2024.448152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2024.471381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2024.518191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2025.063571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2025.254826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2025.283349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2025.338582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2025.370280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2025.390676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2025.414335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2025.440951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2025.467727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2025.482820] LNet: 119461:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2025.499290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2025.521972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2025.549453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2025.573096] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.142@tcp [ 2025.577821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2025.609785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2025.642933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2026.262274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.202.142@tcp [ 2026.507695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2026.546956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2026.574338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2026.606847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2026.645771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2026.683709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2026.720347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2026.770787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2026.793804] LNet: 119578:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2026.818273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2026.860398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2026.902539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2026.937402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2026.963423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2026.990708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2027.528823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2027.718943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2027.750268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2027.790295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2027.830491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2027.882396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2027.936489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2027.970290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2027.988463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2028.013586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2028.037808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2028.071178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2028.108230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2028.144348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2028.176920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2028.474129] LNet: 119741:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2028.477821] LNetError: 119741:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2028.495844] LNet: Removed LNI 192.168.202.42@tcp [ 2028.858690] Key type .llcrypt unregistered [ 2028.864271] Key type ._llcrypt unregistered [ 2042.681936] Key type ._llcrypt registered [ 2042.684260] Key type .llcrypt registered [ 2043.679692] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 03:37:13 (1780385833) [ 2044.040699] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2044.057992] alg: No test for adler32 (adler32-zlib) [ 2044.994450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2045.324236] LNet: 120627:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2045.823533] Key type .llcrypt unregistered [ 2045.825951] Key type ._llcrypt unregistered [ 2051.188336] Key type ._llcrypt registered [ 2051.190452] Key type .llcrypt registered [ 2051.326279] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2051.342144] alg: No test for adler32 (adler32-zlib) [ 2052.225794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2052.241960] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 2057.730413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.142@tcp [ 2065.060286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2065.111065] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2065.138193] LNet: Added LNI 192.168.202.42@tcp1 [8/256/0/180] [ 2066.111434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 2066.611257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.202.142@tcp [ 2066.802749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2066.827454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2066.855273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2066.882822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2066.910223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2066.943776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2066.979571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2067.020631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2067.029703] LNet: 121278:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2067.043427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2067.091927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2067.128799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2067.164426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2067.198060] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2067.242835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2067.901974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2068.095200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2068.129669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2068.160220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2068.192726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2068.226762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2068.258861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2068.283505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2068.313888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2068.330775] LNet: 121395:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2068.342437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2068.365329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2068.386644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2068.413977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2068.437452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2068.461947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2068.998675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.202.142@tcp [ 2069.204792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2069.228431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2069.252818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2069.284612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2069.309227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2069.338964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2069.369318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2069.398151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2069.412598] LNet: 121512:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2069.441723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2069.475826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2069.509028] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2069.557502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2069.586546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2069.623561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2070.200620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2070.416333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2070.441905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2070.474215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2070.509121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2070.537684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2070.566283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2070.591529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2070.620290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2070.658020] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2070.687142] LNet: 2 peer NIs in recovery (showing 2): 192.168.202.142@tcp, 192.168.202.142@tcp1 [ 2070.731292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2070.780105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2070.822507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2070.859952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2070.896320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2075.122149] LNet: 121710:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2075.126985] LNetError: 121710:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2076.191277] LNet: Removed LNI 192.168.202.42@tcp [ 2076.193665] LNetError: 121710:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2076.215915] LNet: Removed LNI 192.168.202.42@tcp1 [ 2076.548605] Key type .llcrypt unregistered [ 2076.558426] Key type ._llcrypt unregistered [ 2090.068970] Key type ._llcrypt registered [ 2090.071262] Key type .llcrypt registered [ 2091.380877] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 03:38:01 (1780385881) [ 2096.910089] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 03:38:07 (1780385887) [ 2102.576987] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 03:38:12 (1780385892) [ 2107.738672] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 03:38:18 (1780385898) [ 2112.564188] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 03:38:22 (1780385902) [ 2117.506300] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 03:38:27 (1780385907) [ 2122.790520] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 03:38:33 (1780385913) [ 2128.062926] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 03:38:38 (1780385918) [ 2133.622520] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 03:38:43 (1780385923) [ 2138.321960] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 03:38:48 (1780385928) [ 2142.891805] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 03:38:53 (1780385933) [ 2147.856775] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 03:38:58 (1780385938) [ 2152.785703] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 03:39:03 (1780385943) [ 2157.597797] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 03:39:08 (1780385948) [ 2162.740818] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 03:39:13 (1780385953) [ 2168.087212] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 03:39:18 (1780385958) [ 2173.801430] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 03:39:24 (1780385964) [ 2179.380468] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 03:39:29 (1780385969) [ 2179.805871] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2179.835101] alg: No test for adler32 (adler32-zlib) [ 2180.777784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2180.906114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2180.935133] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 2182.031608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2182.065426] LNet: 132417:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2182.070529] LNetError: 132417:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2182.095069] LNet: Removed LNI 192.168.202.42@tcp [ 2182.120338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2182.201799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2182.232918] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 2188.136843] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 03:39:38 (1780385978) [ 2193.599554] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 03:39:43 (1780385983) [ 2199.517612] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 03:39:49 (1780385989) [ 2205.857908] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 03:39:56 (1780385996) [ 2211.046916] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 03:40:01 (1780386001) [ 2216.635925] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 03:40:06 (1780386006) [ 2221.974590] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 03:40:12 (1780386012) [ 2227.586392] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 03:40:17 (1780386017) [ 2233.257466] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 03:40:23 (1780386023) [ 2238.455429] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 03:40:28 (1780386028) [ 2243.695589] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 03:40:33 (1780386033) [ 2249.440924] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 03:40:39 (1780386039) [ 2254.904240] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 2037 sec ========== 03:40:44 (1780386044) [ 2255.104534] LNet: 139880:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2255.108542] LNetError: 139880:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2255.124111] LNet: Removed LNI 192.168.202.42@tcp [ 2255.680447] Key type .llcrypt unregistered [ 2255.682456] Key type ._llcrypt unregistered [ 2267.513294] Key type ._llcrypt registered [ 2267.515510] Key type .llcrypt registered [ 2267.913845] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2267.942865] alg: No test for adler32 (adler32-zlib) [ 2268.995848] Lustre: Lustre: Build Version: 2.17.53_57_g7d1c020 [ 2269.202117] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 2270.903325] Key type lgssc registered [ 2272.017034] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2322.667883] Lustre: Mounted lustre-client [ 2327.310296] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2333.864685] Lustre: Unmounted lustre-client [ 2350.783744] Key type lgssc unregistered [ 2351.003427] LNet: 142803:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2351.007647] LNetError: 142803:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2351.025131] LNet: Removed LNI 192.168.202.42@tcp [ 2351.536188] Key type .llcrypt unregistered [ 2351.539397] Key type ._llcrypt unregistered